Statistical physics analysis of machine learning models

• Autor(a) principal: Veiga, Rodrigo Soares
• Data de Publicação: 2022
• Tipo de documento: Tese
• Idioma: eng
• Título da fonte: Biblioteca Digital de Teses e Dissertações da USP
• Texto Completo: https://www.teses.usp.br/teses/disponiveis/43/43134/tde-17082022-084404/
• topic: Aprendizado computacional; Bayesian inference; High-dimensional statistics; Inferência Bayesiana; Machine learning; Mecânica estatística; Métodos estatísticos para aprendizagem; Neural networks; Redes neurais; Statistical physics; Stochastic gradient descent

This thesis presents three main contributions to the understanding of machine learning models by making use of statistical physics tools. First, we investigate the possible relation between the renormalisation group and restricted Boltzmann machines trained with two-dimensional ferromagnetic Ising data, pointing out possible misleadings in preliminary proposals to explicitly construct this bridge. Secondly, we examine the convergence behaviour of stochastic gradient descent in high-dimensional two-layer neural networks. By building up on classic statistical physics approaches and extending them to a broad range of learning rate, time scales, and hidden layer width, we construct a phase diagram describing the various learning scenarios arising in the high-dimensional setting. We also discuss the trade-off between learning rate and hidden layer width, which has been crucial in the recent mean-field theories. Thirdly, we study both Bayes-optimal and empirical risk minimization generalisation errors of the multi-class teacher-student perceptron. We characterise a first-order phase transition arising in the Bayes-optimal performance for Rademacher teacher weights and observe that, for Gaussian teachers, regularised cross-entropy minimisation can yield to close-to-optimal performance.