Characterization of automated machine learning fitness landscapes

• Autor(a) principal: Cristiano Guimarães Pimenta
• Data de Publicação: 2023
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Repositório Institucional da UFMG
• Texto Completo: http://hdl.handle.net/1843/62093; https://orcid.org/0000-0003-2809-8663
• topic: Fitness landscape analysis; Automated machine learning; Search spaces; Optimization; Computação – Teses; Aprendizado do computador – Teses; Otimização combinatória - Teses; Fitness landscape – Teses

Automated Machine Learning (AutoML) aims at automatically selecting and configuring complete machine learning pipelines without requiring deep user expertise. AutoML methods utilize a search space of possible solutions and try to find the best pipeline for a given learning problem. However, there is little knowledge about the characteristics of such spaces and how they relate to the performance of search methods. One way of exploring them is using Fitness Landscape Analysis (FLA), a technique commonly used to describe the landscape of combinatorial optimization problems. This work adapts classic FLA measures, such as Neutrality, Fitness Distance Correlation (FDC) and Correlation Length, to the context of the complex fitness landscape generated by AutoML search spaces, which include discrete, continuous, categorical and conditional variables, regardless of the methods used to explore the search spaces. It also evaluates how the characteristics of the landscape affect the performance of two AutoML methods based on Bayesian optimization: Tree-structured Parzen Estimator (TPE) and Sequential Model-based Algorithm Configuration (SMAC). In order to use FLA in the context of AutoML, we propose a tree-based representation for machine learning pipelines that is able to capture their semantics, a neighborhood definition based on a mutation operator, and a semantic distance metric between pipelines. Neutrality analyses suggest that larger landscapes tend to have more areas of equal or nearly equal fitness values, a feature that can improve the ability of TPE to explore the search space and find good solutions. Larger search spaces tend to be more rugged, as indicated by the Correlation Length measure, and are often more challenging for the optimizers. FDC proved to be a weak measure in describing problem difficulty. Furthermore, using local optima to calculate FDC can lead to very different results when compared to using the global optimum, which is usually unfeasible to calculate for AutoML problems. On the other hand, SMAC’s performance seems less affected by changes in the characteristics of the landscape.