Block size, parallelism and predictive performance: finding the sweet spot in distributed learning

• Autor(a) principal: Oliveira, Filipe
• Data de Publicação: 2023
• Outros Autores: Carneiro, Davide Rua, Guimarães, Miguel, Oliveira, Óscar, Novais, Paulo
• Tipo de documento: Artigo
• Idioma: eng
• Título da fonte: Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)
• Texto Completo: https://hdl.handle.net/1822/89520
• topic: Distributed file system; Distributed machine learning; Hadoop; Machine learning

As distributed and multi-organization Machine Learning emerges, new challenges must be solved, such as diverse and low-quality data or real-time delivery. In this paper, we use a distributed learning environment to analyze the relationship between block size, parallelism, and predictor quality. Specifically, the goal is to find the optimum block size and the best heuristic to create distributed Ensembles. We evaluated three different heuristics and five block sizes on four publicly available datasets. Results show that using fewer but better base models matches or outperforms a standard Random Forest, and that 32 MB is the best block size.