TY - GEN
T1 - Using data flow-based coverage criteria for black-box integration testing of distributed software systems
AU - Hellhake, Dominik
AU - Schmid, Tobias
AU - Wagner, Stefan
N1 - Publisher Copyright:
© 2019 IEEE.
PY - 2019/4
Y1 - 2019/4
N2 - Modern automotive E/E systems are implemented as distributed real-time software systems. The constantly growing complexity of safety-relevant software functions leads to an increased importance of testing during system integration of such systems. Systematic metrics are required to guide the testing process during system integration by providing coverage measures and stopping criteria but few studied approaches exist. For this purpose, we introduce a data-flow based observation scheme which captures the interplay behavior of involved ECUs during test execution and failure occurrences. In addition, we introduce a data flow-based coverage criterion designed for black box integration. By applying the observation scheme to test cases and associated faults found during execution, we first analyze similarities in data flow coverage. By further analyzing the data flow of failures, that slipped through the phase of system integration testing, we evaluate the usefulness of test gaps identified by using the suggested coverage criterion. We found major differences in the usage of data flow between undetected failures and existing test cases. In addition, we found that for the studied system under test the occurrence of failures is not necessarily a direct consequence of the test execution due to functional dependencies and side effects. Overall, these findings highlight the potential and limitations of data flow-based measures to be formalized as coverage or stopping criteria for the integration testing of distributed software systems.
AB - Modern automotive E/E systems are implemented as distributed real-time software systems. The constantly growing complexity of safety-relevant software functions leads to an increased importance of testing during system integration of such systems. Systematic metrics are required to guide the testing process during system integration by providing coverage measures and stopping criteria but few studied approaches exist. For this purpose, we introduce a data-flow based observation scheme which captures the interplay behavior of involved ECUs during test execution and failure occurrences. In addition, we introduce a data flow-based coverage criterion designed for black box integration. By applying the observation scheme to test cases and associated faults found during execution, we first analyze similarities in data flow coverage. By further analyzing the data flow of failures, that slipped through the phase of system integration testing, we evaluate the usefulness of test gaps identified by using the suggested coverage criterion. We found major differences in the usage of data flow between undetected failures and existing test cases. In addition, we found that for the studied system under test the occurrence of failures is not necessarily a direct consequence of the test execution due to functional dependencies and side effects. Overall, these findings highlight the potential and limitations of data flow-based measures to be formalized as coverage or stopping criteria for the integration testing of distributed software systems.
KW - Automotive software systems
KW - Behavior observation
KW - Black box integration testing
KW - Data flow coverage
KW - Distributed software systems
KW - System integration testing
UR - http://www.scopus.com/inward/record.url?scp=85068003977&partnerID=8YFLogxK
U2 - 10.1109/ICST.2019.00051
DO - 10.1109/ICST.2019.00051
M3 - Conference contribution
AN - SCOPUS:85068003977
T3 - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
SP - 420
EP - 429
BT - Proceedings - 2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 12th IEEE International Conference on Software Testing, Verification and Validation, ICST 2019
Y2 - 22 April 2019 through 27 April 2019
ER -