Sebastian Ruland, Lars Luthmann, Johannes Bürdek, Sascha Lity, Thomas Thüm, Malte Lochau, and Márcio Ribeiro

Recent research on quality assurance (QA) of configurable software systems (e.g., software product lines) proposes different analysis strategies to cope with the inherent complexity caused by the well-known combinatorial-explosion problem. Those strategies aim at improving efficiency of QA techniques like software testing as compared to brute-force configuration-by-configuration analysis. Sampling constitutes one of the most established strategies, defining criteria for selecting a drastically reduced, yet sufficiently diverse subset of software configurations considered during QA. However, finding generally accepted measures for assessing the impact of sample-based analysis on the effectiveness of QA techniques is still an open issue. We address this problem by lifting concepts from single-software mutation testing to configurable software. Our framework incorporates a rich collection of mutation operators for product lines implemented in C to measure mutation scores of samples, including a novel family-based technique for product-line mutation detection. Our experimental results gained from applying our tool implementation to a collection of subject systems confirms the widely-accepted assumption that pairwise sampling constitutes the most reasonable efficiency/effectiveness trade-off for sample-based product-line testing.

We performed an experimental evaluation to illustrate the potential and feasability of our approach. The central parts of our replication package can be found here:

VIM

VIM (v8.1) is a new implementation of the UNIX editor VI. Many of the classes and functions in VIM are highly-configurable by a variety of different features.

Source + Featuremodels: VIM

BusyBox

BusyBox (v1.24.2) is a reimplementation of standard Unix tools for systems with limited resources written in C. Many of the tools implemented in BusyBox consist of families of program variants, which are configurable by selection of features.

Source + Featuremodels: BusyBox

Synthetic

We further consider synthetically generated PL implementations in our experiments. To this end, we utilize the tool SiMPOSE2 for reverse-engineering PL implementation from a set of program variants artificially generated from an existing nonconfigurable program

Source + Featuremodels: Synthetic

In the following, we will explain how our experimental results can be reproduced. Our experimental results were produced on a Windows 10 machine with JDK1.8 and eclipse as IDE.

The tool has a number of additional dependencies.

• FeatureIDE: https://featureide.github.io/
• Windows 10 Linux Subsystem: https://docs.microsoft.com/en-us/windows/wsl/install-win10

To reproduce the experimental results:

1. Extract the Zip with the source and import the projects into eclipse
2. Extract the Zip with CPAchecker and import the project into eclipse
3. Build CPAchecker
4. Build all projects
5. Go to MutationtestRunner->src->runner->RunnerConfig.java and adjust the Settings
   1. Programname: name of the case study you want to reproduce the results for
   2. splPath: the Folder the case study is in
   3. cpaCheckerPath: the folder the CPAchecker sourcecode is in
6. Run TestProgramGenerationRunner to reproduce results.

Build CPAchecker:

1. Run build.xml in CPAchecker as Ant build.
2. Build CPAchecker project
3. If the project cannot be build because of the source file "AutomatonParser.java" run CPAchecker->-src->org.sosy_lab.cpacecker->cpa->automaton->automatonbuild.xml as Ant build and refresh the folder. Build the project again.
4. If the project cannot be build because of some source files from de.uni_freiburg... add cpachecker->lib->java->runtime->smtinterpol to cpachecker classpass