System regression test prioritization in factory automation: Relating functional system tests to the tested code using field data

After changes to a system in factory automation, testers are under high time pressure to ensure that changes did not introduce new faults, so-called regressions. This is done using regression testing by re-performing system test, i.e. functional test cases relating to the complete automated production system (aPS), to identify whether previously tested functionality is still fulfilling the requirements. Up to now, system regression testing is exclusively performed manually, heavily relying on the ability and stress resistance of the tester. This results in fluctuating regression test quality and unnecessarily prolonged test processes due to choosing and performing irrelevant test cases. In this paper, an approach in which the tester is supported in choosing, prioritizing and performing relevant test cases during regression testing by means of gathering and analyzing field data acquired during the original execution of the system tests is proposed. By using a guided system testing approach and instrumenting the control program implemented in the IEC 61131-3, it is possible to record relevant data during testing efficiently, in line with real-time constraints that are prevalent in the factory automation domain. The concepts and an evaluation based on two performed experiments using a laboratory case study are presented. The evaluation regarding the performance of the prioritizing approach as well as its run-time properties show promising results for an application in the factory automation domain.