Otimização de dataflows em frameworks de big data por meio do reúso de dados

• Autor(a) principal: Secchim, Gustavo Decarlo Ferreira
• Data de Publicação: 2023
• Tipo de documento: Dissertação
• Idioma: por
• Título da fonte: Biblioteca Digital de Teses e Dissertações do LNCC
• Texto Completo: https://tede.lncc.br/handle/tede/371
• topic: Big data; Data flow; Reúso de dados; Base de dados; Frameworks; CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO::METODOLOGIA E TECNICAS DA COMPUTACAO::BANCO DE DADOS

The use of Big Data frameworks has increased in recent years. These frameworks represent a breakthrough in terms of support for parallel and distributed execution of applications. These applications are often composed of several activities, thus generating a dataflow, which in general processes a large volume of data. As much as the frameworks are optimized to exploit data locality and avoid unnecessary transfers in the distributed environment, such optimizations are focused on isolated executions, i.e. (Do not consider using data from previous runs). This type of data reuse can speed up dataflows, since the data does not need to be processed again if it has already been produced by a previous execution of the same dataflow. This dissertation presents the FORESEE service for sharing data generated in dataflows. We discuss and implement the service using an architecture that allows multiple executions of dataflows to share intermediate results, reducing execution time. We evaluated the approach with real dataflows using open data processing.