Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control
Autor(a) principal: | |
---|---|
Data de Publicação: | 2021 |
Outros Autores: | , |
Tipo de documento: | Artigo |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UFRGS |
Texto Completo: | http://hdl.handle.net/10183/256805 |
Resumo: | In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns. |
id |
UFRGS-2_f933218f39498e4438a2d502075f3621 |
---|---|
oai_identifier_str |
oai:www.lume.ufrgs.br:10183/256805 |
network_acronym_str |
UFRGS-2 |
network_name_str |
Repositório Institucional da UFRGS |
repository_id_str |
|
spelling |
Alegre, Lucas NunesBazzan, Ana Lucia CetertichSilva, Bruno Carreiro da2023-04-07T03:26:12Z20212376-5992http://hdl.handle.net/10183/256805001143089In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns.application/pdfengPeerJ Computer Science. New York : PeerJ, 2021. Vol. 7, (mar. 2021), 20 p.Sistemas multiagentesAprendizado por reforçoInformatica : TransportesMultiagent systemsReinforcement learningTraffic signal controlNon-stationarityQuantifying the impact of non-stationarity in reinforcement learning-based traffic signal controlEstrangeiroinfo:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/openAccessreponame:Repositório Institucional da UFRGSinstname:Universidade Federal do Rio Grande do Sul (UFRGS)instacron:UFRGSTEXT001143089.pdf.txt001143089.pdf.txtExtracted Texttext/plain58613http://www.lume.ufrgs.br/bitstream/10183/256805/2/001143089.pdf.txt310b9c57a7c6332774697ce1ae7ee8f2MD52ORIGINAL001143089.pdfTexto completo (inglês)application/pdf3576109http://www.lume.ufrgs.br/bitstream/10183/256805/1/001143089.pdfe46c67fe048bf5f38de308cf33c44deaMD5110183/2568052023-04-08 03:29:35.852465oai:www.lume.ufrgs.br:10183/256805Repositório de PublicaçõesPUBhttps://lume.ufrgs.br/oai/requestopendoar:2023-04-08T06:29:35Repositório Institucional da UFRGS - Universidade Federal do Rio Grande do Sul (UFRGS)false |
dc.title.pt_BR.fl_str_mv |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
title |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
spellingShingle |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control Alegre, Lucas Nunes Sistemas multiagentes Aprendizado por reforço Informatica : Transportes Multiagent systems Reinforcement learning Traffic signal control Non-stationarity |
title_short |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
title_full |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
title_fullStr |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
title_full_unstemmed |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
title_sort |
Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control |
author |
Alegre, Lucas Nunes |
author_facet |
Alegre, Lucas Nunes Bazzan, Ana Lucia Cetertich Silva, Bruno Carreiro da |
author_role |
author |
author2 |
Bazzan, Ana Lucia Cetertich Silva, Bruno Carreiro da |
author2_role |
author author |
dc.contributor.author.fl_str_mv |
Alegre, Lucas Nunes Bazzan, Ana Lucia Cetertich Silva, Bruno Carreiro da |
dc.subject.por.fl_str_mv |
Sistemas multiagentes Aprendizado por reforço Informatica : Transportes |
topic |
Sistemas multiagentes Aprendizado por reforço Informatica : Transportes Multiagent systems Reinforcement learning Traffic signal control Non-stationarity |
dc.subject.eng.fl_str_mv |
Multiagent systems Reinforcement learning Traffic signal control Non-stationarity |
description |
In reinforcement learning (RL), dealing with non-stationarity is a challenging issue. However, some domains such as traffic optimization are inherently non-stationary. Causes for and effects of this are manifold. In particular, when dealing with traffic signal controls, addressing non-stationarity is key since traffic conditions change over time and as a function of traffic control decisions taken in other parts of a network. In this paper we analyze the effects that different sources of non-stationarity have in a network of traffic signals, in which each signal is modeled as a learning agent. More precisely, we study both the effects of changing the context in which an agent learns (e.g., a change in flow rates experienced by it), as well as the effects of reducing agent observability of the true environment state. Partial observability may cause distinct states (in which distinct actions are optimal) to be seen as the same by the traffic signal agents. This, in turn, may lead to sub-optimal performance. We show that the lack of suitable sensors to provide a representative observation of the real state seems to affect the performance more drastically than the changes to the underlying traffic patterns. |
publishDate |
2021 |
dc.date.issued.fl_str_mv |
2021 |
dc.date.accessioned.fl_str_mv |
2023-04-07T03:26:12Z |
dc.type.driver.fl_str_mv |
Estrangeiro info:eu-repo/semantics/article |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
format |
article |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://hdl.handle.net/10183/256805 |
dc.identifier.issn.pt_BR.fl_str_mv |
2376-5992 |
dc.identifier.nrb.pt_BR.fl_str_mv |
001143089 |
identifier_str_mv |
2376-5992 001143089 |
url |
http://hdl.handle.net/10183/256805 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.relation.ispartof.pt_BR.fl_str_mv |
PeerJ Computer Science. New York : PeerJ, 2021. Vol. 7, (mar. 2021), 20 p. |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.format.none.fl_str_mv |
application/pdf |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFRGS instname:Universidade Federal do Rio Grande do Sul (UFRGS) instacron:UFRGS |
instname_str |
Universidade Federal do Rio Grande do Sul (UFRGS) |
instacron_str |
UFRGS |
institution |
UFRGS |
reponame_str |
Repositório Institucional da UFRGS |
collection |
Repositório Institucional da UFRGS |
bitstream.url.fl_str_mv |
http://www.lume.ufrgs.br/bitstream/10183/256805/2/001143089.pdf.txt http://www.lume.ufrgs.br/bitstream/10183/256805/1/001143089.pdf |
bitstream.checksum.fl_str_mv |
310b9c57a7c6332774697ce1ae7ee8f2 e46c67fe048bf5f38de308cf33c44dea |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFRGS - Universidade Federal do Rio Grande do Sul (UFRGS) |
repository.mail.fl_str_mv |
|
_version_ |
1801225085552427008 |