rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka
Autor(a) principal: | |
---|---|
Data de Publicação: | 2022 |
Tipo de documento: | Dissertação |
Idioma: | por |
Título da fonte: | Repositório Institucional Manancial UFSM |
Texto Completo: | http://repositorio.ufsm.br/handle/1/25169 |
Resumo: | Nowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication. |
id |
UFSM-20_0000357d1932e3a942f2575c88b24b3b |
---|---|
oai_identifier_str |
oai:repositorio.ufsm.br:1/25169 |
network_acronym_str |
UFSM-20 |
network_name_str |
Repositório Institucional Manancial UFSM |
repository_id_str |
3913 |
spelling |
2022-07-01T17:51:29Z2022-07-01T17:51:29Z2022-04-11http://repositorio.ufsm.br/handle/1/25169Nowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication.Atualmente, uma das formas de atingir escalabilidade e flexibilidade durante a construção de aplicações complexas é adotando o paradigma de microsserviços. Por meio de microsserviços, aplicações podem ser fragmentadas em processos isolados e independentes que se comunicam entre si. Entretanto, existe uma problemática presente neste paradigma que trata especificamente da comunicação entre os diversos micro processos de uma mesma aplicação. Para a resolução desta problemática, pode-se aplicar o conceito de mensageria do Apache Kafka para realizar a troca de mensagens entre cada serviço. O Apache Kafka é uma plataforma de mensageria e streaming de dados que segue um modelo produtorconsumidor. A arquitetura do Kafka conta com o mecanismo de Reconhecimento Positivo (ack), o qual visa garantir a entrega de mensagens. Apesar de existirem três níveis de configuração para ack, os mesmos apresentam restrições de confiabilidade ou desempenho durante a transmissão de mensagens em redes instáveis, obrigando usuários a priorizarem um destes requisitos. Este trabalho tem por objetivo apresentar o Reliable Ack (rAck), uma configuração para transmissão de mensagens baseada no monitoramento, identificação e recuperação de mensagens em caso de perda. Uma fase experimental foi conduzida a fim de comparar a configuração rAck com os níveis padrões de ack em termos de confiabilidade e desempenho. Os experimentos foram estruturados para realizar transmissões de mensagens com introdução de falhas de rede que resultavam em perda de pacotes e atraso na entrega de pacotes. Nos cenários com perda de pacotes, a configuração rAck demonstrou melhor desempenho frente aos níveis padrões do Kafka. Já nos cenários com atraso na entrega de pacotes, as falhas introduzidas não impactaram os níveis padrões, mantendo desempenhos melhores que configuração rAck. No que tange à confiabilidade, em ambos os cenários a configuração rAck foi capaz de recuperar todas as mensagens perdidas. Em contrapartida, os níveis padrões estavam sujeitos a perdas ou duplicações de mensagens.Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPESporUniversidade Federal de Santa MariaCentro de TecnologiaPrograma de Pós-Graduação em Ciência da ComputaçãoUFSMBrasilCiência da ComputaçãoAttribution-NonCommercial-NoDerivatives 4.0 Internationalhttp://creativecommons.org/licenses/by-nc-nd/4.0/info:eu-repo/semantics/openAccessSistemas distribuídosMensageriaApache KafkaDistributed systemsMessagingCNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAOrAck: proposta de configuração para garantia de entrega de mensagens no Apache KafkarAck: proposed of configuration for message delivery guarantee in Apache Kafkainfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesisBarcelos, Patrícia Pitthan de Araújohttp://lattes.cnpq.br/6069105173950277Lima, João Vicente FerreiraGriebler, Dalvan Jairhttp://lattes.cnpq.br/8590992494539057Corrêa, Iago da Cunha100300000007600600600600600a421e37e-ac28-4452-9943-e257a19343cacf6135ca-1cde-45aa-a7fb-573dcc9a0684aec00059-8729-40bd-b989-bf07fcb3bdbd1194dfc2-1299-416f-89e6-cfda59a6ebc6reponame:Repositório Institucional Manancial UFSMinstname:Universidade Federal de Santa Maria (UFSM)instacron:UFSMCC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8805http://repositorio.ufsm.br/bitstream/1/25169/2/license_rdf4460e5956bc1d1639be9ae6146a50347MD52LICENSElicense.txtlicense.txttext/plain; charset=utf-816http://repositorio.ufsm.br/bitstream/1/25169/3/license.txtf8fcb28efb1c8cf0dc096bec902bf4c4MD53ORIGINALDIS_PPGCC_2022_CORRÊA_IAGO.pdfDIS_PPGCC_2022_CORRÊA_IAGO.pdfDissertação de Mestradoapplication/pdf1178102http://repositorio.ufsm.br/bitstream/1/25169/1/DIS_PPGCC_2022_CORR%c3%8aA_IAGO.pdfbfaf4db451ff9cb7ab06e227bda2267fMD511/251692022-07-01 14:51:30.137oai:repositorio.ufsm.br:1/25169Q3JlYXRpdmUgQ29tbW9ucw==Repositório Institucionalhttp://repositorio.ufsm.br/PUBhttp://repositorio.ufsm.br/oai/requestouvidoria@ufsm.bropendoar:39132022-07-01T17:51:30Repositório Institucional Manancial UFSM - Universidade Federal de Santa Maria (UFSM)false |
dc.title.por.fl_str_mv |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
dc.title.alternative.eng.fl_str_mv |
rAck: proposed of configuration for message delivery guarantee in Apache Kafka |
title |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
spellingShingle |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka Corrêa, Iago da Cunha Sistemas distribuídos Mensageria Apache Kafka Distributed systems Messaging CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO |
title_short |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_full |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_fullStr |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_full_unstemmed |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
title_sort |
rAck: proposta de configuração para garantia de entrega de mensagens no Apache Kafka |
author |
Corrêa, Iago da Cunha |
author_facet |
Corrêa, Iago da Cunha |
author_role |
author |
dc.contributor.advisor1.fl_str_mv |
Barcelos, Patrícia Pitthan de Araújo |
dc.contributor.advisor1Lattes.fl_str_mv |
http://lattes.cnpq.br/6069105173950277 |
dc.contributor.referee1.fl_str_mv |
Lima, João Vicente Ferreira |
dc.contributor.referee2.fl_str_mv |
Griebler, Dalvan Jair |
dc.contributor.authorLattes.fl_str_mv |
http://lattes.cnpq.br/8590992494539057 |
dc.contributor.author.fl_str_mv |
Corrêa, Iago da Cunha |
contributor_str_mv |
Barcelos, Patrícia Pitthan de Araújo Lima, João Vicente Ferreira Griebler, Dalvan Jair |
dc.subject.por.fl_str_mv |
Sistemas distribuídos Mensageria Apache Kafka |
topic |
Sistemas distribuídos Mensageria Apache Kafka Distributed systems Messaging CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO |
dc.subject.eng.fl_str_mv |
Distributed systems Messaging |
dc.subject.cnpq.fl_str_mv |
CNPQ::CIENCIAS EXATAS E DA TERRA::CIENCIA DA COMPUTACAO |
description |
Nowadays, one of the ways to achieve scalability and flexibility during implementation of applications is by adopting the microsservices paradigm. Through microservices, applications can be fragmented into isolated and independent processes that communicate with each other. However, a concern present in the microservice paradigm specifically deals with the communication between microservices of a same application. To solve this concern, the Apache Kafka messaging concept can be applied to exchange messages between each service. Apache Kafka is a messaging and data streaming platform that follows a producerconsumer model. Kafka’s architecture relies on the Positive Recognition (ack) mechanism, which aims to guarantee the delivery of messages. Although there are three configuration levels for ack, they present reliability or performance restrictions during the transmission of messages in unstable networks, forcing users to prioritize one of these requirements. This work aims to present Reliable Ack (rAck), a configuration for message transmission based on monitoring, identification, and recovery of messages in case of loss. An experimental phase was conducted to compare the rAck configuration with the standard ack levels in terms of reliability and performance. The experiments were structured to carry out message transmissions with the introduction of network failures that resulted in packet loss and delay in delivery. In the packet loss scenarios, the rAck configuration showed better performance when compared to Kafka’s default levels. In scenarios with delays in the delivery of packages, the introduced failures did not impact the default levels, maintaining better performances than rAck configuration. Regarding reliability, in both scenarios, the rAck configuration was able to recover all lost messages. In contrast, the default levels were subject to message loss or duplication. |
publishDate |
2022 |
dc.date.accessioned.fl_str_mv |
2022-07-01T17:51:29Z |
dc.date.available.fl_str_mv |
2022-07-01T17:51:29Z |
dc.date.issued.fl_str_mv |
2022-04-11 |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://repositorio.ufsm.br/handle/1/25169 |
url |
http://repositorio.ufsm.br/handle/1/25169 |
dc.language.iso.fl_str_mv |
por |
language |
por |
dc.relation.cnpq.fl_str_mv |
100300000007 |
dc.relation.confidence.fl_str_mv |
600 600 600 600 600 |
dc.relation.authority.fl_str_mv |
a421e37e-ac28-4452-9943-e257a19343ca cf6135ca-1cde-45aa-a7fb-573dcc9a0684 aec00059-8729-40bd-b989-bf07fcb3bdbd 1194dfc2-1299-416f-89e6-cfda59a6ebc6 |
dc.rights.driver.fl_str_mv |
Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ info:eu-repo/semantics/openAccess |
rights_invalid_str_mv |
Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/ |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Santa Maria Centro de Tecnologia |
dc.publisher.program.fl_str_mv |
Programa de Pós-Graduação em Ciência da Computação |
dc.publisher.initials.fl_str_mv |
UFSM |
dc.publisher.country.fl_str_mv |
Brasil |
dc.publisher.department.fl_str_mv |
Ciência da Computação |
publisher.none.fl_str_mv |
Universidade Federal de Santa Maria Centro de Tecnologia |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional Manancial UFSM instname:Universidade Federal de Santa Maria (UFSM) instacron:UFSM |
instname_str |
Universidade Federal de Santa Maria (UFSM) |
instacron_str |
UFSM |
institution |
UFSM |
reponame_str |
Repositório Institucional Manancial UFSM |
collection |
Repositório Institucional Manancial UFSM |
bitstream.url.fl_str_mv |
http://repositorio.ufsm.br/bitstream/1/25169/2/license_rdf http://repositorio.ufsm.br/bitstream/1/25169/3/license.txt http://repositorio.ufsm.br/bitstream/1/25169/1/DIS_PPGCC_2022_CORR%c3%8aA_IAGO.pdf |
bitstream.checksum.fl_str_mv |
4460e5956bc1d1639be9ae6146a50347 f8fcb28efb1c8cf0dc096bec902bf4c4 bfaf4db451ff9cb7ab06e227bda2267f |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional Manancial UFSM - Universidade Federal de Santa Maria (UFSM) |
repository.mail.fl_str_mv |
ouvidoria@ufsm.br |
_version_ |
1808854727362871296 |