Industrially Applicable System Regression Test Prioritization in Production Automation

When changes are performed on an automated production system (aPS), new faults can be accidentally introduced into the system, which are called regressions. A common method for finding these faults is regression testing. In most cases, this regression testing process is performed under high time pressure and onsite in a very uncomfortable environment. Until now, there has been no automated support for finding and prioritizing system test cases regarding the fully integrated aPS that are suitable for finding regressions. Thus, the testing technician has to rely on personal intuition and experience, possibly choosing an inappropriate order of test cases, finding regressions at a very late stage of the test run. Using a suitable prioritization, this iterative process of finding and fixing regressions can be streamlined and a lot of time can be saved by executing test cases likely to identify new regressions earlier. Thus, an approach is presented in this paper that uses previously acquired runtime data from past test executions and performs a change identification and impact analysis to prioritize test cases that have a high probability to unveil regressions caused by side effects of a system change. The approach was developed in cooperation with reputable industrial partners active in the field of aPS engineering, ensuring a development in line with industrial requirements. An industrial case study and an expert evaluation were performed, showing promising results. Note to Practitioners - Currently, prioritizing relevant system tests to be executed in case of changes to an automated production system (aPS) is very challenging and depends largely on the intuition and experience of the testing technician. As system tests involve manual operations, testing requires substantial effort. In addition, testing is mostly performed under severe time pressure and in an uncomfortable environment such as onsite at the customer's premises. In this paper, an approach is presented that supports the testing technician in finding and prioritizing available test cases based on previous test executions and change analysis. For practitioners, this approach could streamline regression testing and increase the code quality of aPSs significantly, in particular, for fully integrated aPSs, which have not been the focus of other research so far. As the approach was developed in line with industrial requirements, relevance, and adaptability for industrial use are ensured.