Evaluating testing strategies for configurable software systems

• Autor(a) principal: Fischer Jônatas Ferreira
• Data de Publicação: 2021
• Tipo de documento: Tese
• Idioma: eng
• Título da fonte: Repositório Institucional da UFMG
• Texto Completo: http://hdl.handle.net/1843/38084
• topic: Testing Configurable Systems; Testing Tools for Configurable Software Systems; Feature Interactions Faults; Computação – Teses.; Software – Ferramentas – Testes – Teses.; Sistemas configuráveis – Testes – Teses

Configurable software systems allow developers to maintain a unique platform and address a diversity of deployment contexts and usages. Testing configurable systems is essential because configurations that fail may potentially hurt users and degrade the project reputation. However, testing configurable systems is challenging due to the number of configurations to run with each test suite, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing strategies have been proposed to deal with this challenge, but their potential practical application remains largely unexplored. In fact, existing comparisons of testing strategies do not rely on a uniform dataset of configurable software systems. Based on a large dataset of 30 configurable systems, this thesis compares several strategies for testing configurable systems found by a systematic mapping study on two empirical studies. In the first study, we designed and performed a comparative empirical study of the two main sound testing tools, namely VarexJ and SPLat. We also compare sixteen t-wise testing strategies in a second empirical study. With the experience we have in the empirical studies, we propose a list of ten challenges faced when creating test suites for configurable systems and dealing with a test suite for our dataset systems. Our list includes, for instance, the challenges of testing high coupled classes and of determining metrics for measuring the quality of the test suite. Results of the empirical studies indicate which and when strategies are faster and more effective on identifying faults in configurable software systems. Overall, we believe that practitioners acquire the necessary knowledge to choose a testing strategy that best fits their needs with our results and also benefit from our work, observing our solutions for each challenge.