Time Aware Sigmoid Optimization : a new learning rate scheduling method

Detalhes bibliográficos
Autor(a) principal: LEUCHTENBERG, Pedro Henrique Dreyer
Data de Publicação: 2019
Tipo de documento: Dissertação
Idioma: eng
Título da fonte: Repositório Institucional da UFPE
Texto Completo: https://repositorio.ufpe.br/handle/123456789/38541
Resumo: LEUCHTENBERG, Pedro Henrique Dreyer​, também é conhecido em citações bibliográficas por: ​DREYER, Pedro Henrique
id UFPE_c93de2f176178a418e619f64c6cfe7c6
oai_identifier_str oai:repositorio.ufpe.br:123456789/38541
network_acronym_str UFPE
network_name_str Repositório Institucional da UFPE
repository_id_str 2221
spelling LEUCHTENBERG, Pedro Henrique Dreyerhttp://lattes.cnpq.br/3962047609973443http://lattes.cnpq.br/1244195230407619http://lattes.cnpq.br/4271819510740061ZANCHETTIN, CleberMACÊDO, David Lopes de2020-11-09T13:57:38Z2020-11-09T13:57:38Z2019-09-06LEUCHTENBERG, Pedro Henrique Dreyer​. Time Aware Sigmoid Optimization: a new learning rate scheduling method. 2019. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2019.https://repositorio.ufpe.br/handle/123456789/38541LEUCHTENBERG, Pedro Henrique Dreyer​, também é conhecido em citações bibliográficas por: ​DREYER, Pedro HenriqueThe correct choice of hyperparameters for the training of a deep neural network is a critical step to achieve a good result. Good hyperparameters would give rise to faster training and a lower error rate, while bad choices could make the network not even converge, rendering the whole training process useless. Among all the existing hyperparameters, perhaps the one with the greatest importance is the learning rate, which controls how the weights of a neural network are going to change at each interaction. In that context, by analyzing some theoretical findings in the area of information theory and topology of the loss function in deep learning, the author was able to come up with a new training rate decay method called Training Aware Sigmoid Optimization (TASO), which proposes a dual-phase during training. The proposed method aims to improve training, achieving a better inference performance in a reduced amount of time. A series of tests were done to evaluate this hypothesis, comparing TASO with different training methods such as Adam, ADAGrad, RMSProp, and SGD. Results obtained on three datasets (MNIST, CIFAR10, and CIFAR100) and with three different architectures (Lenet, VGG, and RESNET) have shown that TASO presents, in fact, an overall better performance than the other evaluated methods.CAPESA correta escolha dos hiper-parâmetros para o treinamento de uma rede neural profunda é um passo essencial para obter um bom resultado. Bons hiper-parâmetros vãolevar a um treinamento rápido e a uma menor taxa de erro, enquanto que escolhas ruins podem fazer a rede não convergir, inutilizando todo o processo de treinamento. Dentre todos os hiper-parâmetros existentes, talvez o mais crítico seja a taxa de aprendizagem, que irá controlar a magnitude com qual os pesos da rede neural irá atualizar em cada interação. Nesse contexto, esse trabalho avaliou um novo método de mudança na taxa de aprendizagem denominado Training Aware Sigmoid Optimization (TASO), que propõe uma fase dupla de treinamento. O método proposto tem como objetivo melhorar o treinamento, obtendo uma melhor inferência em um menor tempo decorrido. Uma série de testes foi feitas de forma a validar essa hipótese, Comparando TASO com outros métodos de treinamento mais comuns como Adam, ADAGrad, RMSProp, e SGD. Resultados Obtidos em três datasets (MNITS, CIFAR10, e CIFAR100) e três diferentes arquiteturas (Lenet, VGG, e RESNET) mostraram que TASO apresenta uma melhor performance do que os outros métodos avaliados.engUniversidade Federal de PernambucoPrograma de Pos Graduacao em Ciencia da ComputacaoUFPEBrasilAttribution-NonCommercial-NoDerivs 3.0 Brazilhttp://creativecommons.org/licenses/by-nc-nd/3.0/br/info:eu-repo/semantics/openAccessInteligência computacionalAprendizagem de máquinasRedes neurais profundasTaxa de aprendizadoTime Aware Sigmoid Optimization : a new learning rate scheduling methodinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesismestradoreponame:Repositório Institucional da UFPEinstname:Universidade Federal de Pernambuco (UFPE)instacron:UFPEORIGINALDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdfDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdfapplication/pdf1891666https://repositorio.ufpe.br/bitstream/123456789/38541/1/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf4294d8826e7a7ac15a50f27a61839aa9MD51CC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8811https://repositorio.ufpe.br/bitstream/123456789/38541/2/license_rdfe39d27027a6cc9cb039ad269a5db8e34MD52LICENSElicense.txtlicense.txttext/plain; charset=utf-82310https://repositorio.ufpe.br/bitstream/123456789/38541/3/license.txtbd573a5ca8288eb7272482765f819534MD53TEXTDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.txtDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.txtExtracted texttext/plain106745https://repositorio.ufpe.br/bitstream/123456789/38541/4/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.txt9cc86c412e5796362c8ea3d5ca86d733MD54THUMBNAILDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.jpgDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.jpgGenerated Thumbnailimage/jpeg1208https://repositorio.ufpe.br/bitstream/123456789/38541/5/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.jpg0e58c4425a18bcb8f0897901d048478cMD55123456789/385412020-11-10 02:15:51.873oai:repositorio.ufpe.br:123456789/38541TGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKClRvZG8gZGVwb3NpdGFudGUgZGUgbWF0ZXJpYWwgbm8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgKFJJKSBkZXZlIGNvbmNlZGVyLCDDoCBVbml2ZXJzaWRhZGUgRmVkZXJhbCBkZSBQZXJuYW1idWNvIChVRlBFKSwgdW1hIExpY2Vuw6dhIGRlIERpc3RyaWJ1acOnw6NvIE7Do28gRXhjbHVzaXZhIHBhcmEgbWFudGVyIGUgdG9ybmFyIGFjZXNzw612ZWlzIG9zIHNldXMgZG9jdW1lbnRvcywgZW0gZm9ybWF0byBkaWdpdGFsLCBuZXN0ZSByZXBvc2l0w7NyaW8uCgpDb20gYSBjb25jZXNzw6NvIGRlc3RhIGxpY2Vuw6dhIG7Do28gZXhjbHVzaXZhLCBvIGRlcG9zaXRhbnRlIG1hbnTDqW0gdG9kb3Mgb3MgZGlyZWl0b3MgZGUgYXV0b3IuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwoKTGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKCkFvIGNvbmNvcmRhciBjb20gZXN0YSBsaWNlbsOnYSBlIGFjZWl0w6EtbGEsIHZvY8OqIChhdXRvciBvdSBkZXRlbnRvciBkb3MgZGlyZWl0b3MgYXV0b3JhaXMpOgoKYSkgRGVjbGFyYSBxdWUgY29uaGVjZSBhIHBvbMOtdGljYSBkZSBjb3B5cmlnaHQgZGEgZWRpdG9yYSBkbyBzZXUgZG9jdW1lbnRvOwpiKSBEZWNsYXJhIHF1ZSBjb25oZWNlIGUgYWNlaXRhIGFzIERpcmV0cml6ZXMgcGFyYSBvIFJlcG9zaXTDs3JpbyBJbnN0aXR1Y2lvbmFsIGRhIFVGUEU7CmMpIENvbmNlZGUgw6AgVUZQRSBvIGRpcmVpdG8gbsOjbyBleGNsdXNpdm8gZGUgYXJxdWl2YXIsIHJlcHJvZHV6aXIsIGNvbnZlcnRlciAoY29tbyBkZWZpbmlkbyBhIHNlZ3VpciksIGNvbXVuaWNhciBlL291IGRpc3RyaWJ1aXIsIG5vIFJJLCBvIGRvY3VtZW50byBlbnRyZWd1ZSAoaW5jbHVpbmRvIG8gcmVzdW1vL2Fic3RyYWN0KSBlbSBmb3JtYXRvIGRpZ2l0YWwgb3UgcG9yIG91dHJvIG1laW87CmQpIERlY2xhcmEgcXVlIGF1dG9yaXphIGEgVUZQRSBhIGFycXVpdmFyIG1haXMgZGUgdW1hIGPDs3BpYSBkZXN0ZSBkb2N1bWVudG8gZSBjb252ZXJ0w6otbG8sIHNlbSBhbHRlcmFyIG8gc2V1IGNvbnRlw7pkbywgcGFyYSBxdWFscXVlciBmb3JtYXRvIGRlIGZpY2hlaXJvLCBtZWlvIG91IHN1cG9ydGUsIHBhcmEgZWZlaXRvcyBkZSBzZWd1cmFuw6dhLCBwcmVzZXJ2YcOnw6NvIChiYWNrdXApIGUgYWNlc3NvOwplKSBEZWNsYXJhIHF1ZSBvIGRvY3VtZW50byBzdWJtZXRpZG8gw6kgbyBzZXUgdHJhYmFsaG8gb3JpZ2luYWwgZSBxdWUgZGV0w6ltIG8gZGlyZWl0byBkZSBjb25jZWRlciBhIHRlcmNlaXJvcyBvcyBkaXJlaXRvcyBjb250aWRvcyBuZXN0YSBsaWNlbsOnYS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBhIGVudHJlZ2EgZG8gZG9jdW1lbnRvIG7Do28gaW5mcmluZ2Ugb3MgZGlyZWl0b3MgZGUgb3V0cmEgcGVzc29hIG91IGVudGlkYWRlOwpmKSBEZWNsYXJhIHF1ZSwgbm8gY2FzbyBkbyBkb2N1bWVudG8gc3VibWV0aWRvIGNvbnRlciBtYXRlcmlhbCBkbyBxdWFsIG7Do28gZGV0w6ltIG9zIGRpcmVpdG9zIGRlCmF1dG9yLCBvYnRldmUgYSBhdXRvcml6YcOnw6NvIGlycmVzdHJpdGEgZG8gcmVzcGVjdGl2byBkZXRlbnRvciBkZXNzZXMgZGlyZWl0b3MgcGFyYSBjZWRlciDDoApVRlBFIG9zIGRpcmVpdG9zIHJlcXVlcmlkb3MgcG9yIGVzdGEgTGljZW7Dp2EgZSBhdXRvcml6YXIgYSB1bml2ZXJzaWRhZGUgYSB1dGlsaXrDoS1sb3MgbGVnYWxtZW50ZS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBlc3NlIG1hdGVyaWFsIGN1am9zIGRpcmVpdG9zIHPDo28gZGUgdGVyY2Vpcm9zIGVzdMOhIGNsYXJhbWVudGUgaWRlbnRpZmljYWRvIGUgcmVjb25oZWNpZG8gbm8gdGV4dG8gb3UgY29udGXDumRvIGRvIGRvY3VtZW50byBlbnRyZWd1ZTsKZykgU2UgbyBkb2N1bWVudG8gZW50cmVndWUgw6kgYmFzZWFkbyBlbSB0cmFiYWxobyBmaW5hbmNpYWRvIG91IGFwb2lhZG8gcG9yIG91dHJhIGluc3RpdHVpw6fDo28gcXVlIG7Do28gYSBVRlBFLCBkZWNsYXJhIHF1ZSBjdW1wcml1IHF1YWlzcXVlciBvYnJpZ2HDp8O1ZXMgZXhpZ2lkYXMgcGVsbyByZXNwZWN0aXZvIGNvbnRyYXRvIG91IGFjb3Jkby4KCkEgVUZQRSBpZGVudGlmaWNhcsOhIGNsYXJhbWVudGUgbyhzKSBub21lKHMpIGRvKHMpIGF1dG9yIChlcykgZG9zIGRpcmVpdG9zIGRvIGRvY3VtZW50byBlbnRyZWd1ZSBlIG7Do28gZmFyw6EgcXVhbHF1ZXIgYWx0ZXJhw6fDo28sIHBhcmEgYWzDqW0gZG8gcHJldmlzdG8gbmEgYWzDrW5lYSBjKS4KRepositório InstitucionalPUBhttps://repositorio.ufpe.br/oai/requestattena@ufpe.bropendoar:22212020-11-10T05:15:51Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE)false
dc.title.pt_BR.fl_str_mv Time Aware Sigmoid Optimization : a new learning rate scheduling method
title Time Aware Sigmoid Optimization : a new learning rate scheduling method
spellingShingle Time Aware Sigmoid Optimization : a new learning rate scheduling method
LEUCHTENBERG, Pedro Henrique Dreyer
Inteligência computacional
Aprendizagem de máquinas
Redes neurais profundas
Taxa de aprendizado
title_short Time Aware Sigmoid Optimization : a new learning rate scheduling method
title_full Time Aware Sigmoid Optimization : a new learning rate scheduling method
title_fullStr Time Aware Sigmoid Optimization : a new learning rate scheduling method
title_full_unstemmed Time Aware Sigmoid Optimization : a new learning rate scheduling method
title_sort Time Aware Sigmoid Optimization : a new learning rate scheduling method
author LEUCHTENBERG, Pedro Henrique Dreyer
author_facet LEUCHTENBERG, Pedro Henrique Dreyer
author_role author
dc.contributor.authorLattes.pt_BR.fl_str_mv http://lattes.cnpq.br/3962047609973443
dc.contributor.advisorLattes.pt_BR.fl_str_mv http://lattes.cnpq.br/1244195230407619
dc.contributor.advisor-coLattes.pt_BR.fl_str_mv http://lattes.cnpq.br/4271819510740061
dc.contributor.author.fl_str_mv LEUCHTENBERG, Pedro Henrique Dreyer
dc.contributor.advisor1.fl_str_mv ZANCHETTIN, Cleber
dc.contributor.advisor-co1.fl_str_mv MACÊDO, David Lopes de
contributor_str_mv ZANCHETTIN, Cleber
MACÊDO, David Lopes de
dc.subject.por.fl_str_mv Inteligência computacional
Aprendizagem de máquinas
Redes neurais profundas
Taxa de aprendizado
topic Inteligência computacional
Aprendizagem de máquinas
Redes neurais profundas
Taxa de aprendizado
description LEUCHTENBERG, Pedro Henrique Dreyer​, também é conhecido em citações bibliográficas por: ​DREYER, Pedro Henrique
publishDate 2019
dc.date.issued.fl_str_mv 2019-09-06
dc.date.accessioned.fl_str_mv 2020-11-09T13:57:38Z
dc.date.available.fl_str_mv 2020-11-09T13:57:38Z
dc.type.status.fl_str_mv info:eu-repo/semantics/publishedVersion
dc.type.driver.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
status_str publishedVersion
dc.identifier.citation.fl_str_mv LEUCHTENBERG, Pedro Henrique Dreyer​. Time Aware Sigmoid Optimization: a new learning rate scheduling method. 2019. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2019.
dc.identifier.uri.fl_str_mv https://repositorio.ufpe.br/handle/123456789/38541
identifier_str_mv LEUCHTENBERG, Pedro Henrique Dreyer​. Time Aware Sigmoid Optimization: a new learning rate scheduling method. 2019. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2019.
url https://repositorio.ufpe.br/handle/123456789/38541
dc.language.iso.fl_str_mv eng
language eng
dc.rights.driver.fl_str_mv Attribution-NonCommercial-NoDerivs 3.0 Brazil
http://creativecommons.org/licenses/by-nc-nd/3.0/br/
info:eu-repo/semantics/openAccess
rights_invalid_str_mv Attribution-NonCommercial-NoDerivs 3.0 Brazil
http://creativecommons.org/licenses/by-nc-nd/3.0/br/
eu_rights_str_mv openAccess
dc.publisher.none.fl_str_mv Universidade Federal de Pernambuco
dc.publisher.program.fl_str_mv Programa de Pos Graduacao em Ciencia da Computacao
dc.publisher.initials.fl_str_mv UFPE
dc.publisher.country.fl_str_mv Brasil
publisher.none.fl_str_mv Universidade Federal de Pernambuco
dc.source.none.fl_str_mv reponame:Repositório Institucional da UFPE
instname:Universidade Federal de Pernambuco (UFPE)
instacron:UFPE
instname_str Universidade Federal de Pernambuco (UFPE)
instacron_str UFPE
institution UFPE
reponame_str Repositório Institucional da UFPE
collection Repositório Institucional da UFPE
bitstream.url.fl_str_mv https://repositorio.ufpe.br/bitstream/123456789/38541/1/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf
https://repositorio.ufpe.br/bitstream/123456789/38541/2/license_rdf
https://repositorio.ufpe.br/bitstream/123456789/38541/3/license.txt
https://repositorio.ufpe.br/bitstream/123456789/38541/4/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.txt
https://repositorio.ufpe.br/bitstream/123456789/38541/5/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.jpg
bitstream.checksum.fl_str_mv 4294d8826e7a7ac15a50f27a61839aa9
e39d27027a6cc9cb039ad269a5db8e34
bd573a5ca8288eb7272482765f819534
9cc86c412e5796362c8ea3d5ca86d733
0e58c4425a18bcb8f0897901d048478c
bitstream.checksumAlgorithm.fl_str_mv MD5
MD5
MD5
MD5
MD5
repository.name.fl_str_mv Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE)
repository.mail.fl_str_mv attena@ufpe.br
_version_ 1802310587748712448

Registros relacionados