Time Aware Sigmoid Optimization : a new learning rate scheduling method
Autor(a) principal: | |
---|---|
Data de Publicação: | 2019 |
Tipo de documento: | Dissertação |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UFPE |
Texto Completo: | https://repositorio.ufpe.br/handle/123456789/38541 |
Resumo: | LEUCHTENBERG, Pedro Henrique Dreyer, também é conhecido em citações bibliográficas por: DREYER, Pedro Henrique |
id |
UFPE_c93de2f176178a418e619f64c6cfe7c6 |
---|---|
oai_identifier_str |
oai:repositorio.ufpe.br:123456789/38541 |
network_acronym_str |
UFPE |
network_name_str |
Repositório Institucional da UFPE |
repository_id_str |
2221 |
spelling |
LEUCHTENBERG, Pedro Henrique Dreyerhttp://lattes.cnpq.br/3962047609973443http://lattes.cnpq.br/1244195230407619http://lattes.cnpq.br/4271819510740061ZANCHETTIN, CleberMACÊDO, David Lopes de2020-11-09T13:57:38Z2020-11-09T13:57:38Z2019-09-06LEUCHTENBERG, Pedro Henrique Dreyer. Time Aware Sigmoid Optimization: a new learning rate scheduling method. 2019. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2019.https://repositorio.ufpe.br/handle/123456789/38541LEUCHTENBERG, Pedro Henrique Dreyer, também é conhecido em citações bibliográficas por: DREYER, Pedro HenriqueThe correct choice of hyperparameters for the training of a deep neural network is a critical step to achieve a good result. Good hyperparameters would give rise to faster training and a lower error rate, while bad choices could make the network not even converge, rendering the whole training process useless. Among all the existing hyperparameters, perhaps the one with the greatest importance is the learning rate, which controls how the weights of a neural network are going to change at each interaction. In that context, by analyzing some theoretical findings in the area of information theory and topology of the loss function in deep learning, the author was able to come up with a new training rate decay method called Training Aware Sigmoid Optimization (TASO), which proposes a dual-phase during training. The proposed method aims to improve training, achieving a better inference performance in a reduced amount of time. A series of tests were done to evaluate this hypothesis, comparing TASO with different training methods such as Adam, ADAGrad, RMSProp, and SGD. Results obtained on three datasets (MNIST, CIFAR10, and CIFAR100) and with three different architectures (Lenet, VGG, and RESNET) have shown that TASO presents, in fact, an overall better performance than the other evaluated methods.CAPESA correta escolha dos hiper-parâmetros para o treinamento de uma rede neural profunda é um passo essencial para obter um bom resultado. Bons hiper-parâmetros vãolevar a um treinamento rápido e a uma menor taxa de erro, enquanto que escolhas ruins podem fazer a rede não convergir, inutilizando todo o processo de treinamento. Dentre todos os hiper-parâmetros existentes, talvez o mais crítico seja a taxa de aprendizagem, que irá controlar a magnitude com qual os pesos da rede neural irá atualizar em cada interação. Nesse contexto, esse trabalho avaliou um novo método de mudança na taxa de aprendizagem denominado Training Aware Sigmoid Optimization (TASO), que propõe uma fase dupla de treinamento. O método proposto tem como objetivo melhorar o treinamento, obtendo uma melhor inferência em um menor tempo decorrido. Uma série de testes foi feitas de forma a validar essa hipótese, Comparando TASO com outros métodos de treinamento mais comuns como Adam, ADAGrad, RMSProp, e SGD. Resultados Obtidos em três datasets (MNITS, CIFAR10, e CIFAR100) e três diferentes arquiteturas (Lenet, VGG, e RESNET) mostraram que TASO apresenta uma melhor performance do que os outros métodos avaliados.engUniversidade Federal de PernambucoPrograma de Pos Graduacao em Ciencia da ComputacaoUFPEBrasilAttribution-NonCommercial-NoDerivs 3.0 Brazilhttp://creativecommons.org/licenses/by-nc-nd/3.0/br/info:eu-repo/semantics/openAccessInteligência computacionalAprendizagem de máquinasRedes neurais profundasTaxa de aprendizadoTime Aware Sigmoid Optimization : a new learning rate scheduling methodinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesismestradoreponame:Repositório Institucional da UFPEinstname:Universidade Federal de Pernambuco (UFPE)instacron:UFPEORIGINALDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdfDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdfapplication/pdf1891666https://repositorio.ufpe.br/bitstream/123456789/38541/1/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf4294d8826e7a7ac15a50f27a61839aa9MD51CC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8811https://repositorio.ufpe.br/bitstream/123456789/38541/2/license_rdfe39d27027a6cc9cb039ad269a5db8e34MD52LICENSElicense.txtlicense.txttext/plain; charset=utf-82310https://repositorio.ufpe.br/bitstream/123456789/38541/3/license.txtbd573a5ca8288eb7272482765f819534MD53TEXTDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.txtDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.txtExtracted texttext/plain106745https://repositorio.ufpe.br/bitstream/123456789/38541/4/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.txt9cc86c412e5796362c8ea3d5ca86d733MD54THUMBNAILDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.jpgDISSERTAÇÃO Pedro Henrique Dreyer Leuchtenberg.pdf.jpgGenerated Thumbnailimage/jpeg1208https://repositorio.ufpe.br/bitstream/123456789/38541/5/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.jpg0e58c4425a18bcb8f0897901d048478cMD55123456789/385412020-11-10 02:15:51.873oai:repositorio.ufpe.br:123456789/38541TGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKClRvZG8gZGVwb3NpdGFudGUgZGUgbWF0ZXJpYWwgbm8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgKFJJKSBkZXZlIGNvbmNlZGVyLCDDoCBVbml2ZXJzaWRhZGUgRmVkZXJhbCBkZSBQZXJuYW1idWNvIChVRlBFKSwgdW1hIExpY2Vuw6dhIGRlIERpc3RyaWJ1acOnw6NvIE7Do28gRXhjbHVzaXZhIHBhcmEgbWFudGVyIGUgdG9ybmFyIGFjZXNzw612ZWlzIG9zIHNldXMgZG9jdW1lbnRvcywgZW0gZm9ybWF0byBkaWdpdGFsLCBuZXN0ZSByZXBvc2l0w7NyaW8uCgpDb20gYSBjb25jZXNzw6NvIGRlc3RhIGxpY2Vuw6dhIG7Do28gZXhjbHVzaXZhLCBvIGRlcG9zaXRhbnRlIG1hbnTDqW0gdG9kb3Mgb3MgZGlyZWl0b3MgZGUgYXV0b3IuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwoKTGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKCkFvIGNvbmNvcmRhciBjb20gZXN0YSBsaWNlbsOnYSBlIGFjZWl0w6EtbGEsIHZvY8OqIChhdXRvciBvdSBkZXRlbnRvciBkb3MgZGlyZWl0b3MgYXV0b3JhaXMpOgoKYSkgRGVjbGFyYSBxdWUgY29uaGVjZSBhIHBvbMOtdGljYSBkZSBjb3B5cmlnaHQgZGEgZWRpdG9yYSBkbyBzZXUgZG9jdW1lbnRvOwpiKSBEZWNsYXJhIHF1ZSBjb25oZWNlIGUgYWNlaXRhIGFzIERpcmV0cml6ZXMgcGFyYSBvIFJlcG9zaXTDs3JpbyBJbnN0aXR1Y2lvbmFsIGRhIFVGUEU7CmMpIENvbmNlZGUgw6AgVUZQRSBvIGRpcmVpdG8gbsOjbyBleGNsdXNpdm8gZGUgYXJxdWl2YXIsIHJlcHJvZHV6aXIsIGNvbnZlcnRlciAoY29tbyBkZWZpbmlkbyBhIHNlZ3VpciksIGNvbXVuaWNhciBlL291IGRpc3RyaWJ1aXIsIG5vIFJJLCBvIGRvY3VtZW50byBlbnRyZWd1ZSAoaW5jbHVpbmRvIG8gcmVzdW1vL2Fic3RyYWN0KSBlbSBmb3JtYXRvIGRpZ2l0YWwgb3UgcG9yIG91dHJvIG1laW87CmQpIERlY2xhcmEgcXVlIGF1dG9yaXphIGEgVUZQRSBhIGFycXVpdmFyIG1haXMgZGUgdW1hIGPDs3BpYSBkZXN0ZSBkb2N1bWVudG8gZSBjb252ZXJ0w6otbG8sIHNlbSBhbHRlcmFyIG8gc2V1IGNvbnRlw7pkbywgcGFyYSBxdWFscXVlciBmb3JtYXRvIGRlIGZpY2hlaXJvLCBtZWlvIG91IHN1cG9ydGUsIHBhcmEgZWZlaXRvcyBkZSBzZWd1cmFuw6dhLCBwcmVzZXJ2YcOnw6NvIChiYWNrdXApIGUgYWNlc3NvOwplKSBEZWNsYXJhIHF1ZSBvIGRvY3VtZW50byBzdWJtZXRpZG8gw6kgbyBzZXUgdHJhYmFsaG8gb3JpZ2luYWwgZSBxdWUgZGV0w6ltIG8gZGlyZWl0byBkZSBjb25jZWRlciBhIHRlcmNlaXJvcyBvcyBkaXJlaXRvcyBjb250aWRvcyBuZXN0YSBsaWNlbsOnYS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBhIGVudHJlZ2EgZG8gZG9jdW1lbnRvIG7Do28gaW5mcmluZ2Ugb3MgZGlyZWl0b3MgZGUgb3V0cmEgcGVzc29hIG91IGVudGlkYWRlOwpmKSBEZWNsYXJhIHF1ZSwgbm8gY2FzbyBkbyBkb2N1bWVudG8gc3VibWV0aWRvIGNvbnRlciBtYXRlcmlhbCBkbyBxdWFsIG7Do28gZGV0w6ltIG9zIGRpcmVpdG9zIGRlCmF1dG9yLCBvYnRldmUgYSBhdXRvcml6YcOnw6NvIGlycmVzdHJpdGEgZG8gcmVzcGVjdGl2byBkZXRlbnRvciBkZXNzZXMgZGlyZWl0b3MgcGFyYSBjZWRlciDDoApVRlBFIG9zIGRpcmVpdG9zIHJlcXVlcmlkb3MgcG9yIGVzdGEgTGljZW7Dp2EgZSBhdXRvcml6YXIgYSB1bml2ZXJzaWRhZGUgYSB1dGlsaXrDoS1sb3MgbGVnYWxtZW50ZS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBlc3NlIG1hdGVyaWFsIGN1am9zIGRpcmVpdG9zIHPDo28gZGUgdGVyY2Vpcm9zIGVzdMOhIGNsYXJhbWVudGUgaWRlbnRpZmljYWRvIGUgcmVjb25oZWNpZG8gbm8gdGV4dG8gb3UgY29udGXDumRvIGRvIGRvY3VtZW50byBlbnRyZWd1ZTsKZykgU2UgbyBkb2N1bWVudG8gZW50cmVndWUgw6kgYmFzZWFkbyBlbSB0cmFiYWxobyBmaW5hbmNpYWRvIG91IGFwb2lhZG8gcG9yIG91dHJhIGluc3RpdHVpw6fDo28gcXVlIG7Do28gYSBVRlBFLCBkZWNsYXJhIHF1ZSBjdW1wcml1IHF1YWlzcXVlciBvYnJpZ2HDp8O1ZXMgZXhpZ2lkYXMgcGVsbyByZXNwZWN0aXZvIGNvbnRyYXRvIG91IGFjb3Jkby4KCkEgVUZQRSBpZGVudGlmaWNhcsOhIGNsYXJhbWVudGUgbyhzKSBub21lKHMpIGRvKHMpIGF1dG9yIChlcykgZG9zIGRpcmVpdG9zIGRvIGRvY3VtZW50byBlbnRyZWd1ZSBlIG7Do28gZmFyw6EgcXVhbHF1ZXIgYWx0ZXJhw6fDo28sIHBhcmEgYWzDqW0gZG8gcHJldmlzdG8gbmEgYWzDrW5lYSBjKS4KRepositório InstitucionalPUBhttps://repositorio.ufpe.br/oai/requestattena@ufpe.bropendoar:22212020-11-10T05:15:51Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE)false |
dc.title.pt_BR.fl_str_mv |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
title |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
spellingShingle |
Time Aware Sigmoid Optimization : a new learning rate scheduling method LEUCHTENBERG, Pedro Henrique Dreyer Inteligência computacional Aprendizagem de máquinas Redes neurais profundas Taxa de aprendizado |
title_short |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
title_full |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
title_fullStr |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
title_full_unstemmed |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
title_sort |
Time Aware Sigmoid Optimization : a new learning rate scheduling method |
author |
LEUCHTENBERG, Pedro Henrique Dreyer |
author_facet |
LEUCHTENBERG, Pedro Henrique Dreyer |
author_role |
author |
dc.contributor.authorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/3962047609973443 |
dc.contributor.advisorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/1244195230407619 |
dc.contributor.advisor-coLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/4271819510740061 |
dc.contributor.author.fl_str_mv |
LEUCHTENBERG, Pedro Henrique Dreyer |
dc.contributor.advisor1.fl_str_mv |
ZANCHETTIN, Cleber |
dc.contributor.advisor-co1.fl_str_mv |
MACÊDO, David Lopes de |
contributor_str_mv |
ZANCHETTIN, Cleber MACÊDO, David Lopes de |
dc.subject.por.fl_str_mv |
Inteligência computacional Aprendizagem de máquinas Redes neurais profundas Taxa de aprendizado |
topic |
Inteligência computacional Aprendizagem de máquinas Redes neurais profundas Taxa de aprendizado |
description |
LEUCHTENBERG, Pedro Henrique Dreyer, também é conhecido em citações bibliográficas por: DREYER, Pedro Henrique |
publishDate |
2019 |
dc.date.issued.fl_str_mv |
2019-09-06 |
dc.date.accessioned.fl_str_mv |
2020-11-09T13:57:38Z |
dc.date.available.fl_str_mv |
2020-11-09T13:57:38Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.citation.fl_str_mv |
LEUCHTENBERG, Pedro Henrique Dreyer. Time Aware Sigmoid Optimization: a new learning rate scheduling method. 2019. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2019. |
dc.identifier.uri.fl_str_mv |
https://repositorio.ufpe.br/handle/123456789/38541 |
identifier_str_mv |
LEUCHTENBERG, Pedro Henrique Dreyer. Time Aware Sigmoid Optimization: a new learning rate scheduling method. 2019. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2019. |
url |
https://repositorio.ufpe.br/handle/123456789/38541 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
Attribution-NonCommercial-NoDerivs 3.0 Brazil http://creativecommons.org/licenses/by-nc-nd/3.0/br/ info:eu-repo/semantics/openAccess |
rights_invalid_str_mv |
Attribution-NonCommercial-NoDerivs 3.0 Brazil http://creativecommons.org/licenses/by-nc-nd/3.0/br/ |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.publisher.program.fl_str_mv |
Programa de Pos Graduacao em Ciencia da Computacao |
dc.publisher.initials.fl_str_mv |
UFPE |
dc.publisher.country.fl_str_mv |
Brasil |
publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFPE instname:Universidade Federal de Pernambuco (UFPE) instacron:UFPE |
instname_str |
Universidade Federal de Pernambuco (UFPE) |
instacron_str |
UFPE |
institution |
UFPE |
reponame_str |
Repositório Institucional da UFPE |
collection |
Repositório Institucional da UFPE |
bitstream.url.fl_str_mv |
https://repositorio.ufpe.br/bitstream/123456789/38541/1/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf https://repositorio.ufpe.br/bitstream/123456789/38541/2/license_rdf https://repositorio.ufpe.br/bitstream/123456789/38541/3/license.txt https://repositorio.ufpe.br/bitstream/123456789/38541/4/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.txt https://repositorio.ufpe.br/bitstream/123456789/38541/5/DISSERTA%c3%87%c3%83O%20Pedro%20Henrique%20Dreyer%20Leuchtenberg.pdf.jpg |
bitstream.checksum.fl_str_mv |
4294d8826e7a7ac15a50f27a61839aa9 e39d27027a6cc9cb039ad269a5db8e34 bd573a5ca8288eb7272482765f819534 9cc86c412e5796362c8ea3d5ca86d733 0e58c4425a18bcb8f0897901d048478c |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE) |
repository.mail.fl_str_mv |
attena@ufpe.br |
_version_ |
1802310587748712448 |