Automatic evaluation systems for programming tasks (graders) are complex software systems. Automatically performed regression tests can uncover potential defects in the grader at low cost. In this paper, we will describe how sample solutions can act as input data for automatic regression tests. Thus, we are concerned with presenting a solution to a software engineering problem for e-learning systems.

We consider which features of the ProFormA task format can be used for automatic regression testing and propose extensions to the format. The paper describes the exemplary implementation of an automatic black box system test for the Graja auto-evaluator and addresses, among other things, the design of a record-playback procedure, a fuzzy target-actual comparison, and the issue of localizability of detected regressions.