Grants and Contributions:

Grant or Award spanning more than one fiscal year. (2017-2018 to 2022-2023)

In modern software systems "one-size fits all" does not hold because they must be customized to fit the needs of diverse and large populations of users. Variable software refers to this type of software systems that can be configured in a large number of ways, and that stem primarily from the adoption of generator-based techniques, full-blown Software Product Line (SPL) approaches, advanced modularization paradigms, highly-configurable systems, or ad hoc reuse practices collectively called clone-and-own. Testing variable software is specially challenging because of the typically large number of configurations that must be tested, a fact that makes it unfeasible to test each individual configuration.

Combinatorial Interaction Testing (CIT) has been advocated as a paradigm to address this challenge because of the diverse set of algorithms available which can be exploited to compute representative groups of configurations for testing. There are however several open problems that this research program aims to tackle. First, there is a critical lack of publicly-available case studies of variable software on which to empirically evaluate and validate CIT techniques. Second, there is a stark need of a thorough empirical and comparative evaluation of different CIT techniques that have been applied to variable software. Third, there is no adequate tool support for testing of variable software. The current practice is an ad-hoc collection of uncoordinated and incomplete assortment of plug-ins and stand-alone tools.

The objectives for this research program are:
1) Exploit techniques for mining software repositories to gather publicly-available case studies of variable software systems for a thorough empirical evaluation of CIT techniques applied to variable software.
2) Perform a systematic assessment of CIT techniques applied to variable software.
3) Leverage information visualization techniques to better convey the large amount of information present in variable software case studies.
4) Provide robust tool support that allows software engineers to design, implement, and carry out test of variable software.

The first objective will be achieved by studying open source projects available in repositories such as GitHub and developing tools that mine configuration and fault data. For the second objective we will systematically collect and catalog tools for CIT testing of variable software, and evaluate them with the case studies’ data of the first objective. For the third objective we will explore and evaluate different visualization techniques to convey all the distinct types of information that must be considered for making adequate engineering decisions when testing variable software. For the fourth objective, we will develop a framework that allows flexible addition of testing tools and visualization interfaces to provide software engineers with robust tool support of testing tasks.