The automated grader Graja, which is used in teaching to evaluate student Java programs, represents a complex software system. Due to a hardly existing test coverage by module and integration tests, the guarantee of error-free functionality with regard to further development is not guaranteed. Since Graja is based on the ProFormA task format, the question arises to what extent the sample solutions assumed in the ProFormA task format are suitable for an automated regression testing mechanism.

The goal of this research is to create a concept for such a regression testing mechanism as well as putting it into practice with the help of a reference implementation as a Graja extension. The resulting testing mechanism operates by behavioral recording and behavioral comparison, thus converting the evaluation behavior of a sample solution observed in Graja into a test case. In the test phase, a comparison of the target behavior of a test case and the actual observed behavior of a sample solution takes place subsequently. The differences of this comparison are to be treated as potential regressions, since they represent a semantic change of the evaluation result.

In order to implement this behavioral comparison robustly and with as few false alarms as possible, the data models used in Graja were investigated for suitability with respect to a behavioral recording. In addition, a data recording with a subset of the sample solutions took place. Subsequently, an analysis of this raw data was performed with the goal of detecting potential noise within the recordings. This ultimately allowed a strategy to be developed for noise reduction within the behavioral recordings.

Finally, a data model was developed that allows the regressions detected through behavioral matching to be represented in an understandable and localizable manner. The automated regression testing mechanism created by this thesis thus provides a basis for ensuring proper evaluation functionality within the Graja development process. By detecting regressions using the sample solutions, changes to Graja can now be conscientiously adopted into a production environment.