Nequivack: Assessing Mutation Score Confidence

Dominik Holling, Sebastian Banescu, Marco Probst, Ana Petrovska, Alexander Pretschner

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

15 Scopus citations

Abstract

The mutation score is defined as the number of killed mutants divided by the number of non-equivalent mutants. However, whether a mutant is equivalent to the original program is undecidable in general. Thus, even when improving a test suite, a mutant score assessing this test suite may become worse during the development of a system, because of equivalent mutants introduced during mutant creation. This is a fundamental problem. Using static analysis and symbolic execution, we show how to establish non-equivalence or "don't know" among mutants. If the number of don't knows is small, this is a good indicator that a computed mutation score actually reflects its above definition. We can therefore have an increased confidence that mutation score trends correspond to actual improvements of a test suite's quality, and are not overly polluted by equivalent mutants. Using a set of 14 representative unit size programs, we show that for some, but not all, of these programs, the above confidence can indeed be established. We also evaluate the reproducibility, efficiency and effectiveness of our Nequivack tool. Our findings are that reproducibility is completely given. A single mutant analysis can be performed within 3 seconds on average, which is efficient for practical and industrial applications.

Original languageEnglish
Title of host publicationProceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages152-161
Number of pages10
ISBN (Electronic)9781509018260
DOIs
StatePublished - 1 Aug 2016
Event9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016 - Chicago, United States
Duration: 10 Apr 201615 Apr 2016

Publication series

NameProceedings - 2016 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016

Conference

Conference9th IEEE International Conference on Software Testing, Verification and Validation Workshops, ICSTW 2016
Country/TerritoryUnited States
CityChicago
Period10/04/1615/04/16

Keywords

  • equivalent mutant
  • mutant score confidence
  • mutation score
  • non-equivalence checking

Fingerprint

Dive into the research topics of 'Nequivack: Assessing Mutation Score Confidence'. Together they form a unique fingerprint.

Cite this