Assessing irace for automated machine and deep learning in computer vision

• Autor(a) principal: Vieira, Carlos Eduardo Morais
• Data de Publicação: 2021
• Tipo de documento: Dissertação
• Idioma: por
• Título da fonte: Repositório Institucional da UFRN
• Texto Completo: https://repositorio.ufrn.br/handle/123456789/43771
• topic: Automated machine learning; Algorithm configuration; Computer vision; Deep learning; Neural architectural search

Automated machine learning (AutoML) is a field of great interest to both industry and academia. AutoML has allowed developers working on machine learning (ML) applications to achieve satisfactory results with little to no ML expertise. More recently, AutoML tools focused on deep learning (DL) models have proven especially useful to applications where domain-specific algorithms are predominant, as in computer vision (CV) tasks. Still, AutoML tools focused on simpler ML pipelines remain a relevant alternative, since DL models have high computational resource requirements and offer reduced model interpretability. Among the main AutoML approaches, AutoML based on algorithm configurators (AC) is commonly used to produce simpler pipelines, whereas neural architecture search (NAS) is used to produce deep learning models. These two approaches also intersect, since an AC can be used as a NAS algorithm. In this work, we study the application of the irace AC to both these AutoML methods. The irace configurator has been successfully applied to design effective algorithms for optimization problems, but it has not yet been applied to AutoML. Our assessment is performed in two stages. First, we propose an irace-based AutoML tool to produce simple and effective ML pipelines. The tool is dubbed iSklearn, for which we define a configuration space and setup. We demonstrate that iSklearn is able to produce effective ML pipelines using irace as its AC, with comparable performance to more complex ensembles produced by AutoSklearn, an established configuration-based AutoML tool. Moreover, we show the benefits of using the configuration space and setup proposed for iSklearn, even when coupled with another AC. In the second part of our work, we assess irace as a NAS algorithm. To do so, we evaluate irace on NAS-Bench-101, a recent NAS benchmark for the CIFAR-10 CV dataset. We benchmark irace not only through final-quality assessment, but also as to anytime performance through a bi-objective formulation. Results demonstrate that irace can be used as a NAS algorithm, obtaining comparable results to the best NAS algorithms included in NAS-Bench-101 in terms of final quality. However, further work is required to improve its anytime performance in this context. Finally, we discuss other design choices made in the NAS-Bench-101 benchmark, showing how they affect the relative performance of NAS algorithms, and provide guidelines for improving the assessment of NAS algorithms through the use of NAS-Bench-101.