Development and generalization of a reinforcement learning model for the pump scheduling problem
Autor(a) principal: | |
---|---|
Data de Publicação: | 2023 |
Tipo de documento: | Dissertação |
Idioma: | eng |
Título da fonte: | Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
Texto Completo: | http://hdl.handle.net/10773/40879 |
Resumo: | Optimization techniques are used as efficient strategies for the operation of Water Supply Systems (WSS). However, with the increase in complexity of the WSS’s network, the computational resources required by optimization methods also increases, potentially compromising optimal operation in real-time. This work presents a Reinforcement Learning (RL) model as an alternative to traditional optimization techniques for solving the Pump Scheduling Problem (PSP). The model is made of a Deep Q-Learning Network (DQN) agent and an environment where EPANET 2.0 simulates the hydraulic behaviour of water networks. This study aims at analysing the RL agent’s capability in arriving to a near optimal solution in the PSP, testing several energy tariffs and consumption demand patterns. While the computational cost of training the agent is significantly higher than that of finding optimal strategies using traditional optimisation algorithms, once trained the model is capable of providing near-optimal solutions almost immediately. A state features analysis is done where different configurations of both state features and rewards are tested. The results of the RL model are satisfactory by giving information of both demand and tariff patterns, while limiting the number of pump starts for the time window. A generalization analysis is made as well. The RL model achieved an average accuracy of 94% when trained with multiple tariff and consumption patterns for 60k seconds, compared to the results obtained using a non-linear programming optimizer. Additionally, the RL model used only 1% of the CPU time after training. |
id |
RCAP_fb6bf3353e046a6305bbab51425e4f09 |
---|---|
oai_identifier_str |
oai:ria.ua.pt:10773/40879 |
network_acronym_str |
RCAP |
network_name_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository_id_str |
7160 |
spelling |
Development and generalization of a reinforcement learning model for the pump scheduling problemWSSDRLDQNPump scheduling problemEPANETOptimizationOptimization techniques are used as efficient strategies for the operation of Water Supply Systems (WSS). However, with the increase in complexity of the WSS’s network, the computational resources required by optimization methods also increases, potentially compromising optimal operation in real-time. This work presents a Reinforcement Learning (RL) model as an alternative to traditional optimization techniques for solving the Pump Scheduling Problem (PSP). The model is made of a Deep Q-Learning Network (DQN) agent and an environment where EPANET 2.0 simulates the hydraulic behaviour of water networks. This study aims at analysing the RL agent’s capability in arriving to a near optimal solution in the PSP, testing several energy tariffs and consumption demand patterns. While the computational cost of training the agent is significantly higher than that of finding optimal strategies using traditional optimisation algorithms, once trained the model is capable of providing near-optimal solutions almost immediately. A state features analysis is done where different configurations of both state features and rewards are tested. The results of the RL model are satisfactory by giving information of both demand and tariff patterns, while limiting the number of pump starts for the time window. A generalization analysis is made as well. The RL model achieved an average accuracy of 94% when trained with multiple tariff and consumption patterns for 60k seconds, compared to the results obtained using a non-linear programming optimizer. Additionally, the RL model used only 1% of the CPU time after training.Técnicas de otimização são utilizadas como estratégias eficientes para a operação de Sistemas de Abastecimento de Água (SAA). No entanto, com o aumento da complexidade da rede do SAA, os recursos computacionais necessários para métodos de otimização também aumentam, comprometendo potencialmente a operação ideal em tempo real. Este trabalho apresenta um modelo de Reinforcement Learning (RL) como uma alternativa às técnicas tradicionais de otimização para resolver o Problema de Agendamento de Bombas (PAB). O modelo é composto por um agente de Deep Q-learning Network (DQN) e um ambiente onde o EPANET 2.0 simula o comportamento hidráulico da rede de água. Este estudo visa analisar a capacidade do agente RL em chegar a uma solução quase ótima no PAB, testando várias tarifas de energia e padrões de demanda de consumo. Embora o custo computacional do treinamento do agente seja significativamente maior do que o necessário para encontrar estratégias ótimas usando algoritmos tradicionais de otimização, uma vez treinado, o modelo é capaz de fornecer soluções quase ótimas quase imediatamente. Uma análise de características de estado é realizada, testando diferentes configurações de características de estado e recompensas. Os resultados do modelo de RL são satisfatórios, fornecendo informações sobre padrões de consumo e tarifas e limitando o número de acionamentos da bomba para a janela de tempo. Uma análise de generalização também é realizada. Quando treinado com padrões variados de tarifas e consumo por 60 mil segundos, o modelo de RL alcançaa uma precisão média de 94%, em comparação com os resultados obtidos usando um otimizador de programação não linear. Além disso, o modelo de RL utiliza apenas 1% do tempo da CPU após o treinamento.2024-02-27T12:06:52Z2023-12-04T00:00:00Z2023-12-04info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesisapplication/pdfhttp://hdl.handle.net/10773/40879engBorges, Guilherme Simõesinfo:eu-repo/semantics/openAccessreponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãoinstacron:RCAAP2024-03-04T01:45:58Zoai:ria.ua.pt:10773/40879Portal AgregadorONGhttps://www.rcaap.pt/oai/openaireopendoar:71602024-03-20T03:12:36.067063Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãofalse |
dc.title.none.fl_str_mv |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
title |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
spellingShingle |
Development and generalization of a reinforcement learning model for the pump scheduling problem Borges, Guilherme Simões WSS DRL DQN Pump scheduling problem EPANET Optimization |
title_short |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
title_full |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
title_fullStr |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
title_full_unstemmed |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
title_sort |
Development and generalization of a reinforcement learning model for the pump scheduling problem |
author |
Borges, Guilherme Simões |
author_facet |
Borges, Guilherme Simões |
author_role |
author |
dc.contributor.author.fl_str_mv |
Borges, Guilherme Simões |
dc.subject.por.fl_str_mv |
WSS DRL DQN Pump scheduling problem EPANET Optimization |
topic |
WSS DRL DQN Pump scheduling problem EPANET Optimization |
description |
Optimization techniques are used as efficient strategies for the operation of Water Supply Systems (WSS). However, with the increase in complexity of the WSS’s network, the computational resources required by optimization methods also increases, potentially compromising optimal operation in real-time. This work presents a Reinforcement Learning (RL) model as an alternative to traditional optimization techniques for solving the Pump Scheduling Problem (PSP). The model is made of a Deep Q-Learning Network (DQN) agent and an environment where EPANET 2.0 simulates the hydraulic behaviour of water networks. This study aims at analysing the RL agent’s capability in arriving to a near optimal solution in the PSP, testing several energy tariffs and consumption demand patterns. While the computational cost of training the agent is significantly higher than that of finding optimal strategies using traditional optimisation algorithms, once trained the model is capable of providing near-optimal solutions almost immediately. A state features analysis is done where different configurations of both state features and rewards are tested. The results of the RL model are satisfactory by giving information of both demand and tariff patterns, while limiting the number of pump starts for the time window. A generalization analysis is made as well. The RL model achieved an average accuracy of 94% when trained with multiple tariff and consumption patterns for 60k seconds, compared to the results obtained using a non-linear programming optimizer. Additionally, the RL model used only 1% of the CPU time after training. |
publishDate |
2023 |
dc.date.none.fl_str_mv |
2023-12-04T00:00:00Z 2023-12-04 2024-02-27T12:06:52Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://hdl.handle.net/10773/40879 |
url |
http://hdl.handle.net/10773/40879 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.format.none.fl_str_mv |
application/pdf |
dc.source.none.fl_str_mv |
reponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação instacron:RCAAP |
instname_str |
Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
instacron_str |
RCAAP |
institution |
RCAAP |
reponame_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
collection |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository.name.fl_str_mv |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
repository.mail.fl_str_mv |
|
_version_ |
1799137775073624064 |