Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka
Autor(a) principal: | |
---|---|
Data de Publicação: | 2022 |
Tipo de documento: | Dissertação |
Idioma: | por |
Título da fonte: | Manancial - Repositório Digital da UFSM |
Texto Completo: | http://repositorio.ufsm.br/handle/1/25169 |
Resumo: | Nowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication. |
id |
UFSM_0000357d1932e3a942f2575c88b24b3b |
---|---|
oai_identifier_str |
oai:repositorio.ufsm.br:1/25169 |
network_acronym_str |
UFSM |
network_name_str |
Manancial - Repositório Digital da UFSM |
repository_id_str |
|
spelling |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache KafkaRack: proposed of configuration for message delivery guarantee in Apache KafkaSistemas distribuídosMensageriaApache KafkaDistributed systemsMessagingCNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAONowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication.Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPESAtualmente, uma das formas de atingir escalabilidade e flexibilidade durante a construção de aplicações complexas é adotando o paradigma de microsserviços. Por meio de microsserviços, aplicações podem ser fragmentadas em processos isolados e independentes que se comunicam entre si. Entretanto, existe uma problemática presente neste paradigma que trata especificamente da comunicação entre os diversos micro processos de uma mesma aplicação. Para a resolução desta problemática, pode-se aplicar o conceito de mensageria do Apache Kafka para realizar a troca de mensagens entre cada serviço. O Apache Kafka é uma plataforma de mensageria e streaming de dados que segue um modelo produtorconsumidor. A arquitetura do Kafka conta com o mecanismo de Reconhecimento Positivo (ack), o qual visa garantir a entrega de mensagens. Apesar de existirem três níveis de configuração para ack, os mesmos apresentam restrições de confiabilidade ou desempenho durante a transmissão de mensagens em redes instáveis, obrigando usuários a priorizarem um destes requisitos. Este trabalho tem por objetivo apresentar o Reliable Ack (rAck), uma configuração para transmissão de mensagens baseada no monitoramento, identificação e recuperação de mensagens em caso de perda. Uma fase experimental foi conduzida a fim de comparar a configuração rAck com os níveis padrões de ack em termos de confiabilidade e desempenho. Os experimentos foram estruturados para realizar transmissões de mensagens com introdução de falhas de rede que resultavam em perda de pacotes e atraso na entrega de pacotes. Nos cenários com perda de pacotes, a configuração rAck demonstrou melhor desempenho frente aos níveis padrões do Kafka. Já nos cenários com atraso na entrega de pacotes, as falhas introduzidas não impactaram os níveis padrões, mantendo desempenhos melhores que configuração rAck. No que tange à confiabilidade, em ambos os cenários a configuração rAck foi capaz de recuperar todas as mensagens perdidas. Em contrapartida, os níveis padrões estavam sujeitos a perdas ou duplicações de mensagens.Universidade Federal de Santa MariaBrasilCiência da ComputaçãoUFSMPrograma de Pós-Graduação em Ciência da ComputaçãoCentro de TecnologiaBarcelos, Patrícia Pitthan de Araújohttp://lattes.cnpq.br/6069105173950277Lima, João Vicente FerreiraGriebler, Dalvan JairCorrêa, Iago da Cunha2022-07-01T17:51:29Z2022-07-01T17:51:29Z2022-04-11info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesisapplication/pdfhttp://repositorio.ufsm.br/handle/1/25169porAttribution-NonCommercial-NoDerivatives 4.0 Internationalhttp://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessreponame:Manancial - Repositório Digital da UFSMinstname:Universidade Federal de Santa Maria (UFSM)instacron:UFSM2024-07-15T15:43:07Zoai:repositorio.ufsm.br:1/25169Biblioteca Digital de Teses e Dissertaçõeshttps://repositorio.ufsm.br/ONGhttps://repositorio.ufsm.br/oai/requestatendimento.sib@ufsm.br||tedebc@gmail.comopendoar:2024-07-15T15:43:07Manancial - Repositório Digital da UFSM - Universidade Federal de Santa Maria (UFSM)false |
dc.title.none.fl_str_mv |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka Rack: proposed of configuration for message delivery guarantee in Apache Kafka |
title |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
spellingShingle |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka Corrêa, Iago da Cunha Sistemas distribuídos Mensageria Apache Kafka Distributed systems Messaging CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO |
title_short |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_full |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_fullStr |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_full_unstemmed |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_sort |
Rack: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
author |
Corrêa, Iago da Cunha |
author_facet |
Corrêa, Iago da Cunha |
author_role |
author |
dc.contributor.none.fl_str_mv |
Barcelos, Patrícia Pitthan de Araújo http://lattes.cnpq.br/6069105173950277 Lima, João Vicente Ferreira Griebler, Dalvan Jair |
dc.contributor.author.fl_str_mv |
Corrêa, Iago da Cunha |
dc.subject.por.fl_str_mv |
Sistemas distribuídos Mensageria Apache Kafka Distributed systems Messaging CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO |
topic |
Sistemas distribuídos Mensageria Apache Kafka Distributed systems Messaging CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO |
description |
Nowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication. |
publishDate |
2022 |
dc.date.none.fl_str_mv |
2022-07-01T17:51:29Z 2022-07-01T17:51:29Z 2022-04-11 |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://repositorio.ufsm.br/handle/1/25169 |
url |
http://repositorio.ufsm.br/handle/1/25169 |
dc.language.iso.fl_str_mv |
por |
language |
por |
dc.rights.driver.fl_str_mv |
Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ info:eu-repo/semantics/openAccess |
rights_invalid_str_mv |
Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ |
eu_rights_str_mv |
openAccess |
dc.format.none.fl_str_mv |
application/pdf |
dc.publisher.none.fl_str_mv |
Universidade Federal de Santa Maria Brasil Ciência da Computação UFSM Programa de Pós-Graduação em Ciência da Computação Centro de Tecnologia |
publisher.none.fl_str_mv |
Universidade Federal de Santa Maria Brasil Ciência da Computação UFSM Programa de Pós-Graduação em Ciência da Computação Centro de Tecnologia |
dc.source.none.fl_str_mv |
reponame:Manancial - Repositório Digital da UFSM instname:Universidade Federal de Santa Maria (UFSM) instacron:UFSM |
instname_str |
Universidade Federal de Santa Maria (UFSM) |
instacron_str |
UFSM |
institution |
UFSM |
reponame_str |
Manancial - Repositório Digital da UFSM |
collection |
Manancial - Repositório Digital da UFSM |
repository.name.fl_str_mv |
Manancial - Repositório Digital da UFSM - Universidade Federal de Santa Maria (UFSM) |
repository.mail.fl_str_mv |
atendimento.sib@ufsm.br||tedebc@gmail.com |
_version_ |
1805922047405064192 |