Automatic evaluation systems for programming tasks (graders) are complex software systems. Automatically performed regression tests can uncover potential defects in the grader at low cost. In this paper, we will describe how sample solutions can act as input data for automatic regression tests. Thus, we are concerned with presenting a solution to a software engineering problem for e-learning systems.

We consider which features of the ProFormA task format can be used for automatic regression testing and propose extensions to the format. The paper describes the exemplary implementation of an automatic black box system test for the Graja auto-evaluator and addresses, among other things, the design of a record-playback procedure, a fuzzy target-actual comparison, and the issue of localizability of detected regressions.

Software development is a lengthy and complex process in which the quality of the product to be developed plays a decisive role. Because the software development process involves various stakeholders and is an iterative process, quality assurance measures must be taken in order to deliver a product that meets the requirements of the customer. One of these quality assurance measures is software testing, which is used to validate the software in relation to its intended use.

In this paper to the already held presentation the generic term of the software tests does not stand in the foreground, instead an insight into the subset of the regression tests is given. Regression tests differ in their basic mode of operation from the widely used module and integration tests, but are nevertheless a cornerstone of software quality assurance.

The automated grader Graja, which is used in teaching to evaluate student Java programs, represents a complex software system. Due to a hardly existing test coverage by module and integration tests, the guarantee of error-free functionality with regard to further development is not guaranteed. Since Graja is based on the ProFormA task format, the question arises to what extent the sample solutions assumed in the ProFormA task format are suitable for an automated regression testing mechanism.

The goal of this research is to create a concept for such a regression testing mechanism as well as putting it into practice with the help of a reference implementation as a Graja extension. The resulting testing mechanism operates by behavioral recording and behavioral comparison, thus converting the evaluation behavior of a sample solution observed in Graja into a test case. In the test phase, a comparison of the target behavior of a test case and the actual observed behavior of a sample solution takes place subsequently. The differences of this comparison are to be treated as potential regressions, since they represent a semantic change of the evaluation result.

In order to implement this behavioral comparison robustly and with as few false alarms as possible, the data models used in Graja were investigated for suitability with respect to a behavioral recording. In addition, a data recording with a subset of the sample solutions took place. Subsequently, an analysis of this raw data was performed with the goal of detecting potential noise within the recordings. This ultimately allowed a strategy to be developed for noise reduction within the behavioral recordings.

Finally, a data model was developed that allows the regressions detected through behavioral matching to be represented in an understandable and localizable manner. The automated regression testing mechanism created by this thesis thus provides a basis for ensuring proper evaluation functionality within the Graja development process. By detecting regressions using the sample solutions, changes to Graja can now be conscientiously adopted into a production environment.