< Back to previous page

Publication

Comparing mutation coverage against branch coverage in an industrial setting

Journal Contribution - Journal Article

The state-of-the-practice in software development is driven by constant change fueled by continuous integration servers. Such constant change demands for frequent and fully automated tests capable to detect faults immediately upon project build. As the fault detection capability of the test suite becomes so important, modern software development teams continuously monitor the quality of the test suite as well. However, it appears that the state-of-the-practice is reluctant to adopt strong coverage metrics (namely mutation coverage), instead relying on weaker kinds of coverage (namely branch coverage). In this paper, we investigate three reasons that prohibit the adoption of mutation coverage in a continuous integration setting: (1) the difficulty of its integration into the build system, (2) the perception that branch coverage is “good enough”, and (3) the performance overhead during the build. Our investigation is based on a case study involving four open source systems and one industrial system. We demonstrate that mutation coverage reveals additional weaknesses in the test suite compared to branch coverage and that it is able to do so with an acceptable performance overhead during project build.
Journal: International journal on software tools for technology transfer
ISSN: 1433-2779
Volume: 99
Pages: 1 - 24
Publication year:2020
Keywords:A1 Journal article
BOF-keylabel:yes
BOF-publication weight:1
CSS-citation score:1
Authors from:Government
Accessibility:Open