Evaluating testing strategies for configurable software systems
Autor(a) principal: | |
---|---|
Data de Publicação: | 2021 |
Tipo de documento: | Tese |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UFMG |
Texto Completo: | http://hdl.handle.net/1843/38084 |
Resumo: | Configurable software systems allow developers to maintain a unique platform and address a diversity of deployment contexts and usages. Testing configurable systems is essential because configurations that fail may potentially hurt users and degrade the project reputation. However, testing configurable systems is challenging due to the number of configurations to run with each test suite, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing strategies have been proposed to deal with this challenge, but their potential practical application remains largely unexplored. In fact, existing comparisons of testing strategies do not rely on a uniform dataset of configurable software systems. Based on a large dataset of 30 configurable systems, this thesis compares several strategies for testing configurable systems found by a systematic mapping study on two empirical studies. In the first study, we designed and performed a comparative empirical study of the two main sound testing tools, namely VarexJ and SPLat. We also compare sixteen t-wise testing strategies in a second empirical study. With the experience we have in the empirical studies, we propose a list of ten challenges faced when creating test suites for configurable systems and dealing with a test suite for our dataset systems. Our list includes, for instance, the challenges of testing high coupled classes and of determining metrics for measuring the quality of the test suite. Results of the empirical studies indicate which and when strategies are faster and more effective on identifying faults in configurable software systems. Overall, we believe that practitioners acquire the necessary knowledge to choose a testing strategy that best fits their needs with our results and also benefit from our work, observing our solutions for each challenge. |
id |
UFMG_0ca07c26eca4d55ada737f6a9e8e4b63 |
---|---|
oai_identifier_str |
oai:repositorio.ufmg.br:1843/38084 |
network_acronym_str |
UFMG |
network_name_str |
Repositório Institucional da UFMG |
repository_id_str |
|
spelling |
Eduardo Magno Lages Figueiredo Figueiredohttp://lattes.cnpq.br/1265706528850746Marco Túlio de Oliveira ValenteAndré Cavalcante HoraVander Ramos AlvesIvan do Carmo Machadohttp://lattes.cnpq.br/1412888913678183Fischer Jônatas Ferreira2021-09-19T23:59:24Z2021-09-19T23:59:24Z2021-08-13http://hdl.handle.net/1843/38084Configurable software systems allow developers to maintain a unique platform and address a diversity of deployment contexts and usages. Testing configurable systems is essential because configurations that fail may potentially hurt users and degrade the project reputation. However, testing configurable systems is challenging due to the number of configurations to run with each test suite, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing strategies have been proposed to deal with this challenge, but their potential practical application remains largely unexplored. In fact, existing comparisons of testing strategies do not rely on a uniform dataset of configurable software systems. Based on a large dataset of 30 configurable systems, this thesis compares several strategies for testing configurable systems found by a systematic mapping study on two empirical studies. In the first study, we designed and performed a comparative empirical study of the two main sound testing tools, namely VarexJ and SPLat. We also compare sixteen t-wise testing strategies in a second empirical study. With the experience we have in the empirical studies, we propose a list of ten challenges faced when creating test suites for configurable systems and dealing with a test suite for our dataset systems. Our list includes, for instance, the challenges of testing high coupled classes and of determining metrics for measuring the quality of the test suite. Results of the empirical studies indicate which and when strategies are faster and more effective on identifying faults in configurable software systems. Overall, we believe that practitioners acquire the necessary knowledge to choose a testing strategy that best fits their needs with our results and also benefit from our work, observing our solutions for each challenge.Sistemas de software configuráveis permitem que desenvolvedores mantenham uma plataforma única atendendo a uma diversidade de contextos, usos e implantações. Os testes de sistemas configuráveis são essenciais porque as configurações que falham podem prejudicar os usuários e degradar a reputação do projeto. No entanto, testar sistemas configuráveis é muito desafiador devido ao número de configurações a serem executadas em cada teste, levando a uma explosão combinatória do número de configurações e testes. Atualmente, várias estratégias de teste foram propostas para lidar com esse desafio, mas suas potenciais aplicações práticas permanecem amplamente inexploradas. Na verdade, as comparações preliminares existentes de estratégias de testes não visam um conjunto uniforme de sistemas configuráveis. Com base em um grande conjunto de dados de 30 sistemas configuráveis, esta tese compara várias estratégias para testar sistemas configuráveis encontrados por um estudo de mapeamento sistemático. No primeiro estudo, foi projetado e realizado um estudo empírico comparativo com as duas principais ferramentas de teste sólido, chamadas VarexJ e SPLat. Em um segundo estudo empírico foram comparadas dezesseis estratégias de testes pareados. Com a experiência adquirida por meio dos estudos empíricos foi proposta uma lista de dez desafios enfrentados ao criar as suítes de teste para sistemas configuráveis. Ainda, foi relatado como os autores lidaram com as suítes de teste para o conjunto de sistemas descritos nesta tese. A lista proposta inclui, por exemplo, os desafios de testar classes de alto acoplamento e de determinar métricas para medir a qualidade do conjunto de testes. Os resultados dos estudos empíricos indicam quais e quando as estratégias de testes são mais rápidas e eficazes para identificar falhas em sistemas configuráveis. No geral, os autores acreditam que os profissionais podem adquirir o conhecimento necessário por meio dos resultados alcançados, a fim de escolherem uma estratégia de teste que melhor se adapte às suas necessidades e ainda, os profissionais podem se beneficiar com as soluções propostas para cada desafio.CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível SuperiorengUniversidade Federal de Minas GeraisPrograma de Pós-Graduação em Ciência da ComputaçãoUFMGBrasilICX - DEPARTAMENTO DE CIÊNCIA DA COMPUTAÇÃOPrograma Institucional de Internacionalização – CAPES - PrIntComputação – Teses.Software – Ferramentas – Testes – Teses.Sistemas configuráveis – Testes – TesesTesting Configurable SystemsTesting Tools for Configurable Software SystemsFeature Interactions FaultsEvaluating testing strategies for configurable software systemsAvaliação de estratégias de testes para sistemas configuráveis de softwareinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/doctoralThesisinfo:eu-repo/semantics/openAccessreponame:Repositório Institucional da UFMGinstname:Universidade Federal de Minas Gerais (UFMG)instacron:UFMGORIGINALTese_Fischer_FichaCatalografica.pdfTese_Fischer_FichaCatalografica.pdfapplication/pdf9451965https://repositorio.ufmg.br/bitstream/1843/38084/1/Tese_Fischer_FichaCatalografica.pdf0432d24b0b08d4a883d08d10d1b6d47aMD51LICENSElicense.txtlicense.txttext/plain; charset=utf-82118https://repositorio.ufmg.br/bitstream/1843/38084/2/license.txtcda590c95a0b51b4d15f60c9642ca272MD521843/380842021-09-19 20:59:24.631oai:repositorio.ufmg.br:1843/38084TElDRU7Dh0EgREUgRElTVFJJQlVJw4fDg08gTsODTy1FWENMVVNJVkEgRE8gUkVQT1NJVMOTUklPIElOU1RJVFVDSU9OQUwgREEgVUZNRwoKQ29tIGEgYXByZXNlbnRhw6fDo28gZGVzdGEgbGljZW7Dp2EsIHZvY8OqIChvIGF1dG9yIChlcykgb3UgbyB0aXR1bGFyIGRvcyBkaXJlaXRvcyBkZSBhdXRvcikgY29uY2VkZSBhbyBSZXBvc2l0w7NyaW8gSW5zdGl0dWNpb25hbCBkYSBVRk1HIChSSS1VRk1HKSBvIGRpcmVpdG8gbsOjbyBleGNsdXNpdm8gZSBpcnJldm9nw6F2ZWwgZGUgcmVwcm9kdXppciBlL291IGRpc3RyaWJ1aXIgYSBzdWEgcHVibGljYcOnw6NvIChpbmNsdWluZG8gbyByZXN1bW8pIHBvciB0b2RvIG8gbXVuZG8gbm8gZm9ybWF0byBpbXByZXNzbyBlIGVsZXRyw7RuaWNvIGUgZW0gcXVhbHF1ZXIgbWVpbywgaW5jbHVpbmRvIG9zIGZvcm1hdG9zIMOhdWRpbyBvdSB2w61kZW8uCgpWb2PDqiBkZWNsYXJhIHF1ZSBjb25oZWNlIGEgcG9sw610aWNhIGRlIGNvcHlyaWdodCBkYSBlZGl0b3JhIGRvIHNldSBkb2N1bWVudG8gZSBxdWUgY29uaGVjZSBlIGFjZWl0YSBhcyBEaXJldHJpemVzIGRvIFJJLVVGTUcuCgpWb2PDqiBjb25jb3JkYSBxdWUgbyBSZXBvc2l0w7NyaW8gSW5zdGl0dWNpb25hbCBkYSBVRk1HIHBvZGUsIHNlbSBhbHRlcmFyIG8gY29udGXDumRvLCB0cmFuc3BvciBhIHN1YSBwdWJsaWNhw6fDo28gcGFyYSBxdWFscXVlciBtZWlvIG91IGZvcm1hdG8gcGFyYSBmaW5zIGRlIHByZXNlcnZhw6fDo28uCgpWb2PDqiB0YW1iw6ltIGNvbmNvcmRhIHF1ZSBvIFJlcG9zaXTDs3JpbyBJbnN0aXR1Y2lvbmFsIGRhIFVGTUcgcG9kZSBtYW50ZXIgbWFpcyBkZSB1bWEgY8OzcGlhIGRlIHN1YSBwdWJsaWNhw6fDo28gcGFyYSBmaW5zIGRlIHNlZ3VyYW7Dp2EsIGJhY2stdXAgZSBwcmVzZXJ2YcOnw6NvLgoKVm9jw6ogZGVjbGFyYSBxdWUgYSBzdWEgcHVibGljYcOnw6NvIMOpIG9yaWdpbmFsIGUgcXVlIHZvY8OqIHRlbSBvIHBvZGVyIGRlIGNvbmNlZGVyIG9zIGRpcmVpdG9zIGNvbnRpZG9zIG5lc3RhIGxpY2Vuw6dhLiBWb2PDqiB0YW1iw6ltIGRlY2xhcmEgcXVlIG8gZGVww7NzaXRvIGRlIHN1YSBwdWJsaWNhw6fDo28gbsOjbywgcXVlIHNlamEgZGUgc2V1IGNvbmhlY2ltZW50bywgaW5mcmluZ2UgZGlyZWl0b3MgYXV0b3JhaXMgZGUgbmluZ3XDqW0uCgpDYXNvIGEgc3VhIHB1YmxpY2HDp8OjbyBjb250ZW5oYSBtYXRlcmlhbCBxdWUgdm9jw6ogbsOjbyBwb3NzdWkgYSB0aXR1bGFyaWRhZGUgZG9zIGRpcmVpdG9zIGF1dG9yYWlzLCB2b2PDqiBkZWNsYXJhIHF1ZSBvYnRldmUgYSBwZXJtaXNzw6NvIGlycmVzdHJpdGEgZG8gZGV0ZW50b3IgZG9zIGRpcmVpdG9zIGF1dG9yYWlzIHBhcmEgY29uY2VkZXIgYW8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgZGEgVUZNRyBvcyBkaXJlaXRvcyBhcHJlc2VudGFkb3MgbmVzdGEgbGljZW7Dp2EsIGUgcXVlIGVzc2UgbWF0ZXJpYWwgZGUgcHJvcHJpZWRhZGUgZGUgdGVyY2Vpcm9zIGVzdMOhIGNsYXJhbWVudGUgaWRlbnRpZmljYWRvIGUgcmVjb25oZWNpZG8gbm8gdGV4dG8gb3Ugbm8gY29udGXDumRvIGRhIHB1YmxpY2HDp8OjbyBvcmEgZGVwb3NpdGFkYS4KCkNBU08gQSBQVUJMSUNBw4fDg08gT1JBIERFUE9TSVRBREEgVEVOSEEgU0lETyBSRVNVTFRBRE8gREUgVU0gUEFUUk9Dw41OSU8gT1UgQVBPSU8gREUgVU1BIEFHw4pOQ0lBIERFIEZPTUVOVE8gT1UgT1VUUk8gT1JHQU5JU01PLCBWT0PDiiBERUNMQVJBIFFVRSBSRVNQRUlUT1UgVE9ET1MgRSBRVUFJU1FVRVIgRElSRUlUT1MgREUgUkVWSVPDg08gQ09NTyBUQU1Cw4lNIEFTIERFTUFJUyBPQlJJR0HDh8OVRVMgRVhJR0lEQVMgUE9SIENPTlRSQVRPIE9VIEFDT1JETy4KCk8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgZGEgVUZNRyBzZSBjb21wcm9tZXRlIGEgaWRlbnRpZmljYXIgY2xhcmFtZW50ZSBvIHNldSBub21lKHMpIG91IG8ocykgbm9tZXMocykgZG8ocykgZGV0ZW50b3IoZXMpIGRvcyBkaXJlaXRvcyBhdXRvcmFpcyBkYSBwdWJsaWNhw6fDo28sIGUgbsOjbyBmYXLDoSBxdWFscXVlciBhbHRlcmHDp8OjbywgYWzDqW0gZGFxdWVsYXMgY29uY2VkaWRhcyBwb3IgZXN0YSBsaWNlbsOnYS4KRepositório de PublicaçõesPUBhttps://repositorio.ufmg.br/oaiopendoar:2021-09-19T23:59:24Repositório Institucional da UFMG - Universidade Federal de Minas Gerais (UFMG)false |
dc.title.pt_BR.fl_str_mv |
Evaluating testing strategies for configurable software systems |
dc.title.alternative.pt_BR.fl_str_mv |
Avaliação de estratégias de testes para sistemas configuráveis de software |
title |
Evaluating testing strategies for configurable software systems |
spellingShingle |
Evaluating testing strategies for configurable software systems Fischer Jônatas Ferreira Testing Configurable Systems Testing Tools for Configurable Software Systems Feature Interactions Faults Computação – Teses. Software – Ferramentas – Testes – Teses. Sistemas configuráveis – Testes – Teses |
title_short |
Evaluating testing strategies for configurable software systems |
title_full |
Evaluating testing strategies for configurable software systems |
title_fullStr |
Evaluating testing strategies for configurable software systems |
title_full_unstemmed |
Evaluating testing strategies for configurable software systems |
title_sort |
Evaluating testing strategies for configurable software systems |
author |
Fischer Jônatas Ferreira |
author_facet |
Fischer Jônatas Ferreira |
author_role |
author |
dc.contributor.advisor1.fl_str_mv |
Eduardo Magno Lages Figueiredo Figueiredo |
dc.contributor.advisor1Lattes.fl_str_mv |
http://lattes.cnpq.br/1265706528850746 |
dc.contributor.referee1.fl_str_mv |
Marco Túlio de Oliveira Valente |
dc.contributor.referee2.fl_str_mv |
André Cavalcante Hora |
dc.contributor.referee3.fl_str_mv |
Vander Ramos Alves |
dc.contributor.referee4.fl_str_mv |
Ivan do Carmo Machado |
dc.contributor.authorLattes.fl_str_mv |
http://lattes.cnpq.br/1412888913678183 |
dc.contributor.author.fl_str_mv |
Fischer Jônatas Ferreira |
contributor_str_mv |
Eduardo Magno Lages Figueiredo Figueiredo Marco Túlio de Oliveira Valente André Cavalcante Hora Vander Ramos Alves Ivan do Carmo Machado |
dc.subject.por.fl_str_mv |
Testing Configurable Systems Testing Tools for Configurable Software Systems Feature Interactions Faults |
topic |
Testing Configurable Systems Testing Tools for Configurable Software Systems Feature Interactions Faults Computação – Teses. Software – Ferramentas – Testes – Teses. Sistemas configuráveis – Testes – Teses |
dc.subject.other.pt_BR.fl_str_mv |
Computação – Teses. Software – Ferramentas – Testes – Teses. Sistemas configuráveis – Testes – Teses |
description |
Configurable software systems allow developers to maintain a unique platform and address a diversity of deployment contexts and usages. Testing configurable systems is essential because configurations that fail may potentially hurt users and degrade the project reputation. However, testing configurable systems is challenging due to the number of configurations to run with each test suite, leading to a combinatorial explosion in the number of configurations and tests. Currently, several testing strategies have been proposed to deal with this challenge, but their potential practical application remains largely unexplored. In fact, existing comparisons of testing strategies do not rely on a uniform dataset of configurable software systems. Based on a large dataset of 30 configurable systems, this thesis compares several strategies for testing configurable systems found by a systematic mapping study on two empirical studies. In the first study, we designed and performed a comparative empirical study of the two main sound testing tools, namely VarexJ and SPLat. We also compare sixteen t-wise testing strategies in a second empirical study. With the experience we have in the empirical studies, we propose a list of ten challenges faced when creating test suites for configurable systems and dealing with a test suite for our dataset systems. Our list includes, for instance, the challenges of testing high coupled classes and of determining metrics for measuring the quality of the test suite. Results of the empirical studies indicate which and when strategies are faster and more effective on identifying faults in configurable software systems. Overall, we believe that practitioners acquire the necessary knowledge to choose a testing strategy that best fits their needs with our results and also benefit from our work, observing our solutions for each challenge. |
publishDate |
2021 |
dc.date.accessioned.fl_str_mv |
2021-09-19T23:59:24Z |
dc.date.available.fl_str_mv |
2021-09-19T23:59:24Z |
dc.date.issued.fl_str_mv |
2021-08-13 |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/doctoralThesis |
format |
doctoralThesis |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://hdl.handle.net/1843/38084 |
url |
http://hdl.handle.net/1843/38084 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.relation.pt_BR.fl_str_mv |
Programa Institucional de Internacionalização – CAPES - PrInt |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Minas Gerais |
dc.publisher.program.fl_str_mv |
Programa de Pós-Graduação em Ciência da Computação |
dc.publisher.initials.fl_str_mv |
UFMG |
dc.publisher.country.fl_str_mv |
Brasil |
dc.publisher.department.fl_str_mv |
ICX - DEPARTAMENTO DE CIÊNCIA DA COMPUTAÇÃO |
publisher.none.fl_str_mv |
Universidade Federal de Minas Gerais |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFMG instname:Universidade Federal de Minas Gerais (UFMG) instacron:UFMG |
instname_str |
Universidade Federal de Minas Gerais (UFMG) |
instacron_str |
UFMG |
institution |
UFMG |
reponame_str |
Repositório Institucional da UFMG |
collection |
Repositório Institucional da UFMG |
bitstream.url.fl_str_mv |
https://repositorio.ufmg.br/bitstream/1843/38084/1/Tese_Fischer_FichaCatalografica.pdf https://repositorio.ufmg.br/bitstream/1843/38084/2/license.txt |
bitstream.checksum.fl_str_mv |
0432d24b0b08d4a883d08d10d1b6d47a cda590c95a0b51b4d15f60c9642ca272 |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFMG - Universidade Federal de Minas Gerais (UFMG) |
repository.mail.fl_str_mv |
|
_version_ |
1803589525759328256 |