Failure Models for Testing Continuous Controllers

Ranging from temperature control to safety-critical applications, continuous controllers are used in a plethora of applications becoming increasingly complex. In turn, testing continuous control systems also is more complex. Particularly, application-specific manual formal analysis or testing the complete input range becomes infeasible. We present a comprehensive failure-based testing methodology and a respective automated tool for continuous controllers. Our methodology is based on an existing automated approach, testing stability, liveness, smoothness and responsiveness in a single value-response scenario only. We performed a practitioner survey and literature review in the domain revealing the quality criteria steadiness and reliability to be vital for meaningful testing of continuous controllers. In addition, we identified 4 further scenarios including disturbance response for comprehensive testing. We contribute a library of failure models and quality criteria for the automated testing of continuous control systems more complete than in previous approaches. On the grounds of our comprehensive experiments on 9 real-world control systems, our results demonstrate our failure-based testing methodology to provide better worst cases than manual testing (effectiveness) within an adequate time frame (efficiency) for any configuration used in our experiments (reproducibility).