Enhancing deep learning performance using displaced rectifier linear unit
Autor(a) principal: | |
---|---|
Data de Publicação: | 2017 |
Tipo de documento: | Dissertação |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UFPE |
Texto Completo: | https://repositorio.ufpe.br/handle/123456789/28361 |
Resumo: | Recently, deep learning has caused a significant impact on computer vision, speech recognition, and natural language understanding. In spite of the remarkable advances, deep learning recent performance gains have been modest and usually rely on increasing the depth of the models, which often requires more computational resources such as processing time and memory usage. To tackle this problem, we turned our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory currently. In this work, we propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-10 and CIFAR-100, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p<0:05) showed DReLU enhanced the test accuracy obtained by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work shows that it is possible to increase the performance replacing ReLU by an enhanced activation function. |
id |
UFPE_75b5a9240e2a72d48bf9dd460de0de8b |
---|---|
oai_identifier_str |
oai:repositorio.ufpe.br:123456789/28361 |
network_acronym_str |
UFPE |
network_name_str |
Repositório Institucional da UFPE |
repository_id_str |
2221 |
spelling |
MACÊDO, David Lopes dehttp://lattes.cnpq.br/4271819510740061http://lattes.cnpq.br/6321179168854922LUDERMIR, Teresa BernardaZANCHETTIN, Cleber2018-12-28T20:06:59Z2018-12-28T20:06:59Z2017-07-31https://repositorio.ufpe.br/handle/123456789/28361Recently, deep learning has caused a significant impact on computer vision, speech recognition, and natural language understanding. In spite of the remarkable advances, deep learning recent performance gains have been modest and usually rely on increasing the depth of the models, which often requires more computational resources such as processing time and memory usage. To tackle this problem, we turned our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory currently. In this work, we propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-10 and CIFAR-100, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p<0:05) showed DReLU enhanced the test accuracy obtained by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work shows that it is possible to increase the performance replacing ReLU by an enhanced activation function.Recentemente, a aprendizagem profunda tem causado um impacto significante em visão computacional, reconhecimento de voz e compreensão de linguagem natural. Apesar de avanços significativos, recentemente os ganhos em desempenho em aprendizagem profunda tem sido modestos e usualmente dependem do incremento da profundidade dos modelos, o que normalmente requer mais recursos computacionais como tempo de processamento e uso de memória. Para abordar este problema, nós voltamos nossa atenção para o interfuncionamento entre as funções de ativações e a normalização em batch, o qual é praticamente obrigatório atualmente. Neste trabalho, nós propomos a função de ativação Displaced Rectifier Linear Unit (DReLU) a partir da conjectura que estender a função identidade da ReLU para o terceiro quadrante aprimora a compatibilidade com a normalização em batch. Ademais, nós usamos testes estatísticos para comparar o impacto de usar funções de ativação distintas (ReLU, LReLU, PReLU, ELU, and DReLU) na performance da velocidade de treinamento e na acurácia dos testes de modelos estado da arte VGG e Redes Residuais. Estas redes neurais convolucionais foram treinadas no CIFAR-10 e CIFAR-100, as base de dados mais comumente utilizadas em visão computacional para aprendizagem profunda. Os resultados mostraram que DReLU aumentou a velocidade de aprendizagem em todos os modelos e bases de dados. Ademais, avaliações de performance com o uso de testes estatíticos (p<0:05) mostraram que DReLU melhorou a acurácia dos testes apresentados pela ReLU em todos os cenários. Além disso, DReLU apresentou melhor acurácia de testes que qualquer outra função de ativação testada em todos os cenários com uma exceção, no qual esta apresentou a segunda melhor performance. Desta forma, este trabalho mostra que é possível aumentar a performance substituindo a ReLU por uma função de ativação aprimorada.engUniversidade Federal de PernambucoPrograma de Pos Graduacao em Ciencia da ComputacaoUFPEBrasilAttribution-NonCommercial-NoDerivs 3.0 Brazilhttp://creativecommons.org/licenses/by-nc-nd/3.0/br/info:eu-repo/semantics/openAccessInteligência artificialRedes Neurais ConvolucionaisFunções de AtivaçãoEnhancing deep learning performance using displaced rectifier linear unitinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesismestradoreponame:Repositório Institucional da UFPEinstname:Universidade Federal de Pernambuco (UFPE)instacron:UFPETHUMBNAILDISSERTAÇÃO David Lopes de Macêdo.pdf.jpgDISSERTAÇÃO David Lopes de Macêdo.pdf.jpgGenerated Thumbnailimage/jpeg1304https://repositorio.ufpe.br/bitstream/123456789/28361/6/DISSERTA%c3%87%c3%83O%20David%20Lopes%20de%20Mac%c3%aado.pdf.jpg217bac5e37cda86be01529bcdc61aef6MD56ORIGINALDISSERTAÇÃO David Lopes de Macêdo.pdfDISSERTAÇÃO David Lopes de Macêdo.pdfapplication/pdf5139833https://repositorio.ufpe.br/bitstream/123456789/28361/1/DISSERTA%c3%87%c3%83O%20David%20Lopes%20de%20Mac%c3%aado.pdf76748bb5a54eeec793319230c6ba3b30MD51LICENSElicense.txtlicense.txttext/plain; charset=utf-82311https://repositorio.ufpe.br/bitstream/123456789/28361/3/license.txt4b8a02c7f2818eaf00dcf2260dd5eb08MD53CC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8811https://repositorio.ufpe.br/bitstream/123456789/28361/4/license_rdfe39d27027a6cc9cb039ad269a5db8e34MD54TEXTDISSERTAÇÃO David Lopes de Macêdo.pdf.txtDISSERTAÇÃO David Lopes de Macêdo.pdf.txtExtracted texttext/plain131254https://repositorio.ufpe.br/bitstream/123456789/28361/5/DISSERTA%c3%87%c3%83O%20David%20Lopes%20de%20Mac%c3%aado.pdf.txtcec5292cadc3d7dc57ad685ec216d204MD55123456789/283612019-10-25 23:16:42.824oai:repositorio.ufpe.br:123456789/28361TGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKClRvZG8gZGVwb3NpdGFudGUgZGUgbWF0ZXJpYWwgbm8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgKFJJKSBkZXZlIGNvbmNlZGVyLCDDoCBVbml2ZXJzaWRhZGUgRmVkZXJhbCBkZSBQZXJuYW1idWNvIChVRlBFKSwgdW1hIExpY2Vuw6dhIGRlIERpc3RyaWJ1acOnw6NvIE7Do28gRXhjbHVzaXZhIHBhcmEgbWFudGVyIGUgdG9ybmFyIGFjZXNzw612ZWlzIG9zIHNldXMgZG9jdW1lbnRvcywgZW0gZm9ybWF0byBkaWdpdGFsLCBuZXN0ZSByZXBvc2l0w7NyaW8uCgpDb20gYSBjb25jZXNzw6NvIGRlc3RhIGxpY2Vuw6dhIG7Do28gZXhjbHVzaXZhLCBvIGRlcG9zaXRhbnRlIG1hbnTDqW0gdG9kb3Mgb3MgZGlyZWl0b3MgZGUgYXV0b3IuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwoKTGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKCkFvIGNvbmNvcmRhciBjb20gZXN0YSBsaWNlbsOnYSBlIGFjZWl0w6EtbGEsIHZvY8OqIChhdXRvciBvdSBkZXRlbnRvciBkb3MgZGlyZWl0b3MgYXV0b3JhaXMpOgoKYSkgRGVjbGFyYSBxdWUgY29uaGVjZSBhIHBvbMOtdGljYSBkZSBjb3B5cmlnaHQgZGEgZWRpdG9yYSBkbyBzZXUgZG9jdW1lbnRvOwpiKSBEZWNsYXJhIHF1ZSBjb25oZWNlIGUgYWNlaXRhIGFzIERpcmV0cml6ZXMgcGFyYSBvIFJlcG9zaXTDs3JpbyBJbnN0aXR1Y2lvbmFsIGRhIFVGUEU7CmMpIENvbmNlZGUgw6AgVUZQRSBvIGRpcmVpdG8gbsOjbyBleGNsdXNpdm8gZGUgYXJxdWl2YXIsIHJlcHJvZHV6aXIsIGNvbnZlcnRlciAoY29tbyBkZWZpbmlkbyBhIHNlZ3VpciksIGNvbXVuaWNhciBlL291IGRpc3RyaWJ1aXIsIG5vIFJJLCBvIGRvY3VtZW50byBlbnRyZWd1ZSAoaW5jbHVpbmRvIG8gcmVzdW1vL2Fic3RyYWN0KSBlbSBmb3JtYXRvIGRpZ2l0YWwgb3UgcG9yIG91dHJvIG1laW87CmQpIERlY2xhcmEgcXVlIGF1dG9yaXphIGEgVUZQRSBhIGFycXVpdmFyIG1haXMgZGUgdW1hIGPDs3BpYSBkZXN0ZSBkb2N1bWVudG8gZSBjb252ZXJ0w6otbG8sIHNlbSBhbHRlcmFyIG8gc2V1IGNvbnRlw7pkbywgcGFyYSBxdWFscXVlciBmb3JtYXRvIGRlIGZpY2hlaXJvLCBtZWlvIG91IHN1cG9ydGUsIHBhcmEgZWZlaXRvcyBkZSBzZWd1cmFuw6dhLCBwcmVzZXJ2YcOnw6NvIChiYWNrdXApIGUgYWNlc3NvOwplKSBEZWNsYXJhIHF1ZSBvIGRvY3VtZW50byBzdWJtZXRpZG8gw6kgbyBzZXUgdHJhYmFsaG8gb3JpZ2luYWwgZSBxdWUgZGV0w6ltIG8gZGlyZWl0byBkZSBjb25jZWRlciBhIHRlcmNlaXJvcyBvcyBkaXJlaXRvcyBjb250aWRvcyBuZXN0YSBsaWNlbsOnYS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBhIGVudHJlZ2EgZG8gZG9jdW1lbnRvIG7Do28gaW5mcmluZ2Ugb3MgZGlyZWl0b3MgZGUgb3V0cmEgcGVzc29hIG91IGVudGlkYWRlOwpmKSBEZWNsYXJhIHF1ZSwgbm8gY2FzbyBkbyBkb2N1bWVudG8gc3VibWV0aWRvIGNvbnRlciBtYXRlcmlhbCBkbyBxdWFsIG7Do28gZGV0w6ltIG9zIGRpcmVpdG9zIGRlCmF1dG9yLCBvYnRldmUgYSBhdXRvcml6YcOnw6NvIGlycmVzdHJpdGEgZG8gcmVzcGVjdGl2byBkZXRlbnRvciBkZXNzZXMgZGlyZWl0b3MgcGFyYSBjZWRlciDDoApVRlBFIG9zIGRpcmVpdG9zIHJlcXVlcmlkb3MgcG9yIGVzdGEgTGljZW7Dp2EgZSBhdXRvcml6YXIgYSB1bml2ZXJzaWRhZGUgYSB1dGlsaXrDoS1sb3MgbGVnYWxtZW50ZS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBlc3NlIG1hdGVyaWFsIGN1am9zIGRpcmVpdG9zIHPDo28gZGUgdGVyY2Vpcm9zIGVzdMOhIGNsYXJhbWVudGUgaWRlbnRpZmljYWRvIGUgcmVjb25oZWNpZG8gbm8gdGV4dG8gb3UgY29udGXDumRvIGRvIGRvY3VtZW50byBlbnRyZWd1ZTsKZykgU2UgbyBkb2N1bWVudG8gZW50cmVndWUgw6kgYmFzZWFkbyBlbSB0cmFiYWxobyBmaW5hbmNpYWRvIG91IGFwb2lhZG8gcG9yIG91dHJhIGluc3RpdHVpw6fDo28gcXVlIG7Do28gYSBVRlBFLMKgZGVjbGFyYSBxdWUgY3VtcHJpdSBxdWFpc3F1ZXIgb2JyaWdhw6fDtWVzIGV4aWdpZGFzIHBlbG8gcmVzcGVjdGl2byBjb250cmF0byBvdSBhY29yZG8uCgpBIFVGUEUgaWRlbnRpZmljYXLDoSBjbGFyYW1lbnRlIG8ocykgbm9tZShzKSBkbyhzKSBhdXRvciAoZXMpIGRvcyBkaXJlaXRvcyBkbyBkb2N1bWVudG8gZW50cmVndWUgZSBuw6NvIGZhcsOhIHF1YWxxdWVyIGFsdGVyYcOnw6NvLCBwYXJhIGFsw6ltIGRvIHByZXZpc3RvIG5hIGFsw61uZWEgYykuCg==Repositório InstitucionalPUBhttps://repositorio.ufpe.br/oai/requestattena@ufpe.bropendoar:22212019-10-26T02:16:42Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE)false |
dc.title.pt_BR.fl_str_mv |
Enhancing deep learning performance using displaced rectifier linear unit |
title |
Enhancing deep learning performance using displaced rectifier linear unit |
spellingShingle |
Enhancing deep learning performance using displaced rectifier linear unit MACÊDO, David Lopes de Inteligência artificial Redes Neurais Convolucionais Funções de Ativação |
title_short |
Enhancing deep learning performance using displaced rectifier linear unit |
title_full |
Enhancing deep learning performance using displaced rectifier linear unit |
title_fullStr |
Enhancing deep learning performance using displaced rectifier linear unit |
title_full_unstemmed |
Enhancing deep learning performance using displaced rectifier linear unit |
title_sort |
Enhancing deep learning performance using displaced rectifier linear unit |
author |
MACÊDO, David Lopes de |
author_facet |
MACÊDO, David Lopes de |
author_role |
author |
dc.contributor.authorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/4271819510740061 |
dc.contributor.advisorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/6321179168854922 |
dc.contributor.author.fl_str_mv |
MACÊDO, David Lopes de |
dc.contributor.advisor1.fl_str_mv |
LUDERMIR, Teresa Bernarda |
dc.contributor.advisor-co1.fl_str_mv |
ZANCHETTIN, Cleber |
contributor_str_mv |
LUDERMIR, Teresa Bernarda ZANCHETTIN, Cleber |
dc.subject.por.fl_str_mv |
Inteligência artificial Redes Neurais Convolucionais Funções de Ativação |
topic |
Inteligência artificial Redes Neurais Convolucionais Funções de Ativação |
description |
Recently, deep learning has caused a significant impact on computer vision, speech recognition, and natural language understanding. In spite of the remarkable advances, deep learning recent performance gains have been modest and usually rely on increasing the depth of the models, which often requires more computational resources such as processing time and memory usage. To tackle this problem, we turned our attention to the interworking between the activation functions and the batch normalization, which is virtually mandatory currently. In this work, we propose the activation function Displaced Rectifier Linear Unit (DReLU) by conjecturing that extending the identity function of ReLU to the third quadrant enhances compatibility with batch normalization. Moreover, we used statistical tests to compare the impact of using distinct activation functions (ReLU, LReLU, PReLU, ELU, and DReLU) on the learning speed and test accuracy performance of VGG and Residual Networks state-of-the-art models. These convolutional neural networks were trained on CIFAR-10 and CIFAR-100, the most commonly used deep learning computer vision datasets. The results showed DReLU speeded up learning in all models and datasets. Besides, statistical significant performance assessments (p<0:05) showed DReLU enhanced the test accuracy obtained by ReLU in all scenarios. Furthermore, DReLU showed better test accuracy than any other tested activation function in all experiments with one exception, in which case it presented the second best performance. Therefore, this work shows that it is possible to increase the performance replacing ReLU by an enhanced activation function. |
publishDate |
2017 |
dc.date.issued.fl_str_mv |
2017-07-31 |
dc.date.accessioned.fl_str_mv |
2018-12-28T20:06:59Z |
dc.date.available.fl_str_mv |
2018-12-28T20:06:59Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
https://repositorio.ufpe.br/handle/123456789/28361 |
url |
https://repositorio.ufpe.br/handle/123456789/28361 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
Attribution-NonCommercial-NoDerivs 3.0 Brazil http://creativecommons.org/licenses/by-nc-nd/3.0/br/ info:eu-repo/semantics/openAccess |
rights_invalid_str_mv |
Attribution-NonCommercial-NoDerivs 3.0 Brazil http://creativecommons.org/licenses/by-nc-nd/3.0/br/ |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.publisher.program.fl_str_mv |
Programa de Pos Graduacao em Ciencia da Computacao |
dc.publisher.initials.fl_str_mv |
UFPE |
dc.publisher.country.fl_str_mv |
Brasil |
publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFPE instname:Universidade Federal de Pernambuco (UFPE) instacron:UFPE |
instname_str |
Universidade Federal de Pernambuco (UFPE) |
instacron_str |
UFPE |
institution |
UFPE |
reponame_str |
Repositório Institucional da UFPE |
collection |
Repositório Institucional da UFPE |
bitstream.url.fl_str_mv |
https://repositorio.ufpe.br/bitstream/123456789/28361/6/DISSERTA%c3%87%c3%83O%20David%20Lopes%20de%20Mac%c3%aado.pdf.jpg https://repositorio.ufpe.br/bitstream/123456789/28361/1/DISSERTA%c3%87%c3%83O%20David%20Lopes%20de%20Mac%c3%aado.pdf https://repositorio.ufpe.br/bitstream/123456789/28361/3/license.txt https://repositorio.ufpe.br/bitstream/123456789/28361/4/license_rdf https://repositorio.ufpe.br/bitstream/123456789/28361/5/DISSERTA%c3%87%c3%83O%20David%20Lopes%20de%20Mac%c3%aado.pdf.txt |
bitstream.checksum.fl_str_mv |
217bac5e37cda86be01529bcdc61aef6 76748bb5a54eeec793319230c6ba3b30 4b8a02c7f2818eaf00dcf2260dd5eb08 e39d27027a6cc9cb039ad269a5db8e34 cec5292cadc3d7dc57ad685ec216d204 |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE) |
repository.mail.fl_str_mv |
attena@ufpe.br |
_version_ |
1802310672464216064 |