Test orales for systems with complex outputs: the case of TTS systems

• Autor(a) principal: Rafael Alves Paes de Oliveira
• Data de Publicação: 2017
• Tipo de documento: Tese
• Idioma: eng
• Título da fonte: Biblioteca Digital de Teses e Dissertações da USP
• Texto Completo: https://doi.org/10.11606/T.55.2017.tde-13092017-085208

Software testing is one of the most important Software Engineering processes, being the primary activity to check the conformance between the software requirements and its actual behavior. The automation of software testing activities is essential to certify productivity and effectiveness in such activities. Test automation leads testing activities to be conducted under systematic and accurate criteria, raising the chance of testers to reveal faults or inconsistencies. Test oracles are elementary members in software testing automation, being the mechanism responsible for indicating the correctness of software outputs. In testing environments, test oracles can be effectively implemented based on several sources of information about the Software Under Testing (SUT): software specifications, assertions, formal methods (Finite State Machines (FSM), formal specifications, etc, machine-learning methods, and metamorphic relations. Regardless of the implementation strategy, test oracles are vulnerable to false positive/negative verdicts, configuring what the literature describes as the oracle problem. Therefore, test oracles are a non-trivial and challenging object of studies of the software engineering research area. SUTs outputs in unusual formats make it harder the oracle problem. Audio, images, three-dimensional objects, virtual reality environments, complex statistical compositions, etc, are examples of non-trivial output formats. In the software testing context, SUTs with unusual outputs can be called complex-output systems. In this doctorate dissertation, we propose and evaluate a novel test oracle approach for complex-output systems called feature-based test oracles. The purpose of feature-based test oracles is the appropriation of a processing image technique called Content-Based Image Retrieval (CBIR) to collect information from features extracted from the SUTs outputs to compose test oracles. Given a query image, CBIR combines feature extraction and similarity functions to alleviate the problem of searching for digital images in large databases. In previous research, we have integrated CBIR concepts in a testing framework to support the automation of testing activities in processing image systems and systems with Graphical User Interfaces (GUI). In this doctorate dissertation, we extended that framework and its concepts to general complex-output systems, addressing the feature-based test oracle approach. We use Text-To-Speech (TTS) systems to validate empirically our test oracle technique. Through the results of five empirical analyses, three of them conducted in line with problems of a real-world industry TTS system, show the proposed technique is a valuable instrument to automate testing activities and alleviate practitioners efforts on testing complex output systems. We conclude the proposed test oracles are effective because they systematically evaluate the SUTs sensorial output rather than produce verdicts based on subjective specifications. As future work, we plan to conduct investigations towards the reduction of false positives/negatives and the association of the test oracles with machine learning techniques and metamorphic relations.