Squeezed very deep convolutional neural networks for text classification
Main Author: | |
---|---|
Publication Date: | 2020 |
Format: | Master thesis |
Language: | eng |
Source: | Repositório Institucional da UFPE |
Download full: | https://repositorio.ufpe.br/handle/123456789/39490 |
Summary: | Embedding artificial intelligence on constrained platforms has become a trend since the growth of embedded systems and mobile devices, experimented in recent years. Al though constrained platforms do not have enough processing capabilities to train a sophis ticated deep learning model, like Convolutional Neural Network (CNN), they are already capable of performing inference locally by using a previously trained embedded model. This approach enables numerous advantages such as more privacy, smaller response la tency, and no real-time network dependence. Still, the use of a local CNN model on constrained platforms is restricted by its storage size and processing power. Most of the research in CNN has focused on increasing network depth to improve accuracy. In the text classification area, deep models were proposed with excellent performance but rely ing on large architectures with thousands of parameters, and consequently, they require high storage size and processing. One of the models with much renown is the Very Deep Convolutional Neural Networks (VDCNN). In this dissertation, it is proposed an archi tectural modification in the VDCNN model to reduce its storage size while keeping its performance. In this optimization process, the impacts of using Temporal Depthwise Sep arable Convolutions and Global Average Pooling in the network are evaluated regarding parameters, storage size, dedicated hardware dependence, and accuracy. The proposed Squeezed Very Deep Convolutional Neural Networks (SVDCNN) model is between 10x and 20x smaller than the original version, depending on the network depth, maintain ing a maximum disk size of 6MB. Regarding accuracy, the network experiences a loss between 0.1% and 1.0% in the accuracy performance while obtains lower latency over non-dedicated hardware and higher inference time ratio compared to the baseline model. |
id |
UFPE_e0e19d692cfc0ccf624df2b79927b81b |
---|---|
oai_identifier_str |
oai:repositorio.ufpe.br:123456789/39490 |
network_acronym_str |
UFPE |
network_name_str |
Repositório Institucional da UFPE |
repository_id_str |
2221 |
spelling |
SANTOS, Luã Lázaro Jesus doshttp://lattes.cnpq.br/7216467413729634http://lattes.cnpq.br/1244195230407619ZANCHETTIN, Cleber2021-03-26T15:50:24Z2021-03-26T15:50:24Z2020-02-14SANTOS, Luã Lázaro Jesus dos. Squeezed very deep convolutional neural networks for text classification. 2020. Dissertação (Mestrado em Ciência da Computação) - Universidade Federal de Pernambuco, Recife, 2020.https://repositorio.ufpe.br/handle/123456789/39490Embedding artificial intelligence on constrained platforms has become a trend since the growth of embedded systems and mobile devices, experimented in recent years. Al though constrained platforms do not have enough processing capabilities to train a sophis ticated deep learning model, like Convolutional Neural Network (CNN), they are already capable of performing inference locally by using a previously trained embedded model. This approach enables numerous advantages such as more privacy, smaller response la tency, and no real-time network dependence. Still, the use of a local CNN model on constrained platforms is restricted by its storage size and processing power. Most of the research in CNN has focused on increasing network depth to improve accuracy. In the text classification area, deep models were proposed with excellent performance but rely ing on large architectures with thousands of parameters, and consequently, they require high storage size and processing. One of the models with much renown is the Very Deep Convolutional Neural Networks (VDCNN). In this dissertation, it is proposed an archi tectural modification in the VDCNN model to reduce its storage size while keeping its performance. In this optimization process, the impacts of using Temporal Depthwise Sep arable Convolutions and Global Average Pooling in the network are evaluated regarding parameters, storage size, dedicated hardware dependence, and accuracy. The proposed Squeezed Very Deep Convolutional Neural Networks (SVDCNN) model is between 10x and 20x smaller than the original version, depending on the network depth, maintain ing a maximum disk size of 6MB. Regarding accuracy, the network experiences a loss between 0.1% and 1.0% in the accuracy performance while obtains lower latency over non-dedicated hardware and higher inference time ratio compared to the baseline model.CNPqEmbarcar inteligência artificial em plataformas com restrições de desempenho tem se tornado uma tendência desde o crescimento no uso de sistemas embarcados e dispositivos móveis, presenciado nos últimos anos. Apesar de sistemas com restrições de desempenho não terem capacidade de processamento suficiente para treinar modelos complexos, como as Redes Neurais Convolucionais (RNC), eles já são capazes de realizar sua inferência utilizando um modelo embarcado previamente treinado. Essa abordagem oferece diversas vantagens, tais como maior privacidade, menor latência de resposta e a não dependên cia de conexão com a internet em tempo real. De todo modo, o uso de um modelo de RNC em dispositivos com restrições de desempenho é condicionado ao seu tamanho de armazenamento e poder de processamento. Muitas das pesquisas em RNC tem focado em aumentar a profundidade da rede para melhorar sua acurácia. No campo de classificação de texto, modelos profundos apresentam excelente performance, mas se baseiam em ar quiteturas grandes, com milhares de parêmetros, e consequentemente, alto requisito de armazenamento e processamento. Um dos modelos com bastante destaque é o Very Deep Convolutional Neural Networks (VDCNN). Nesta dissertação, é proposta a modificação da estrutura do modelo VDCNN para reduzir seu tamanho de armazenamento mantendo sua performance. Neste processo de otimização, são avaliados os impactos do uso de Depthwise Separable Convolutions e Global Average Pooling na arquitetura da rede, considerando a quantidade de parâmetros, tamanho de armazenamento, dependência de hardware dedi cado e acurácia. O modelo proposto, Squeezed Very Deep Convolutional Neural Networks (SVDCNN), é entre 10 e 20 vezes menor do que sua versão original, dependendo da pro fundidade da rede utilizada, mantendo um tamanho de armazenamento máximo de 6MB. Com relação à acurácia, o modelo experimenta uma perda entre 0.1% e 1.0% na perfo mance de classificação enquanto obtém menor latência em hardware não-dedicado e maior quociente de tempo de inferência comparado com o modelo base.engUniversidade Federal de PernambucoPrograma de Pos Graduacao em Ciencia da ComputacaoUFPEBrasilhttp://creativecommons.org/licenses/by-nc-nd/3.0/br/info:eu-repo/semantics/openAccessInteligência computacionalRedes neurais convolucionaisSqueezed very deep convolutional neural networks for text classificationinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesismestradoreponame:Repositório Institucional da UFPEinstname:Universidade Federal de Pernambuco (UFPE)instacron:UFPEORIGINALDISSERTAÇÃO Luã Lázaro Jesus dos Santos.pdfDISSERTAÇÃO Luã Lázaro Jesus dos Santos.pdfapplication/pdf7556198https://repositorio.ufpe.br/bitstream/123456789/39490/1/DISSERTA%c3%87%c3%83O%20Lu%c3%a3%20L%c3%a1zaro%20Jesus%20dos%20Santos.pdf3434fa0bd6d5becdabb64182b4854997MD51TEXTDISSERTAÇÃO Luã Lázaro Jesus dos Santos.pdf.txtDISSERTAÇÃO Luã Lázaro Jesus dos Santos.pdf.txtExtracted texttext/plain107636https://repositorio.ufpe.br/bitstream/123456789/39490/4/DISSERTA%c3%87%c3%83O%20Lu%c3%a3%20L%c3%a1zaro%20Jesus%20dos%20Santos.pdf.txt80b183d7220a4ee84bd1bb804d6c1e89MD54THUMBNAILDISSERTAÇÃO Luã Lázaro Jesus dos Santos.pdf.jpgDISSERTAÇÃO Luã Lázaro Jesus dos Santos.pdf.jpgGenerated Thumbnailimage/jpeg1245https://repositorio.ufpe.br/bitstream/123456789/39490/5/DISSERTA%c3%87%c3%83O%20Lu%c3%a3%20L%c3%a1zaro%20Jesus%20dos%20Santos.pdf.jpg9510b15fa1a8ab077d51ad1c211563eeMD55CC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8811https://repositorio.ufpe.br/bitstream/123456789/39490/2/license_rdfe39d27027a6cc9cb039ad269a5db8e34MD52LICENSElicense.txtlicense.txttext/plain; charset=utf-82310https://repositorio.ufpe.br/bitstream/123456789/39490/3/license.txtbd573a5ca8288eb7272482765f819534MD53123456789/394902021-03-27 02:15:41.698oai:repositorio.ufpe.br:123456789/39490TGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKClRvZG8gZGVwb3NpdGFudGUgZGUgbWF0ZXJpYWwgbm8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgKFJJKSBkZXZlIGNvbmNlZGVyLCDDoCBVbml2ZXJzaWRhZGUgRmVkZXJhbCBkZSBQZXJuYW1idWNvIChVRlBFKSwgdW1hIExpY2Vuw6dhIGRlIERpc3RyaWJ1acOnw6NvIE7Do28gRXhjbHVzaXZhIHBhcmEgbWFudGVyIGUgdG9ybmFyIGFjZXNzw612ZWlzIG9zIHNldXMgZG9jdW1lbnRvcywgZW0gZm9ybWF0byBkaWdpdGFsLCBuZXN0ZSByZXBvc2l0w7NyaW8uCgpDb20gYSBjb25jZXNzw6NvIGRlc3RhIGxpY2Vuw6dhIG7Do28gZXhjbHVzaXZhLCBvIGRlcG9zaXRhbnRlIG1hbnTDqW0gdG9kb3Mgb3MgZGlyZWl0b3MgZGUgYXV0b3IuCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwoKTGljZW7Dp2EgZGUgRGlzdHJpYnVpw6fDo28gTsOjbyBFeGNsdXNpdmEKCkFvIGNvbmNvcmRhciBjb20gZXN0YSBsaWNlbsOnYSBlIGFjZWl0w6EtbGEsIHZvY8OqIChhdXRvciBvdSBkZXRlbnRvciBkb3MgZGlyZWl0b3MgYXV0b3JhaXMpOgoKYSkgRGVjbGFyYSBxdWUgY29uaGVjZSBhIHBvbMOtdGljYSBkZSBjb3B5cmlnaHQgZGEgZWRpdG9yYSBkbyBzZXUgZG9jdW1lbnRvOwpiKSBEZWNsYXJhIHF1ZSBjb25oZWNlIGUgYWNlaXRhIGFzIERpcmV0cml6ZXMgcGFyYSBvIFJlcG9zaXTDs3JpbyBJbnN0aXR1Y2lvbmFsIGRhIFVGUEU7CmMpIENvbmNlZGUgw6AgVUZQRSBvIGRpcmVpdG8gbsOjbyBleGNsdXNpdm8gZGUgYXJxdWl2YXIsIHJlcHJvZHV6aXIsIGNvbnZlcnRlciAoY29tbyBkZWZpbmlkbyBhIHNlZ3VpciksIGNvbXVuaWNhciBlL291IGRpc3RyaWJ1aXIsIG5vIFJJLCBvIGRvY3VtZW50byBlbnRyZWd1ZSAoaW5jbHVpbmRvIG8gcmVzdW1vL2Fic3RyYWN0KSBlbSBmb3JtYXRvIGRpZ2l0YWwgb3UgcG9yIG91dHJvIG1laW87CmQpIERlY2xhcmEgcXVlIGF1dG9yaXphIGEgVUZQRSBhIGFycXVpdmFyIG1haXMgZGUgdW1hIGPDs3BpYSBkZXN0ZSBkb2N1bWVudG8gZSBjb252ZXJ0w6otbG8sIHNlbSBhbHRlcmFyIG8gc2V1IGNvbnRlw7pkbywgcGFyYSBxdWFscXVlciBmb3JtYXRvIGRlIGZpY2hlaXJvLCBtZWlvIG91IHN1cG9ydGUsIHBhcmEgZWZlaXRvcyBkZSBzZWd1cmFuw6dhLCBwcmVzZXJ2YcOnw6NvIChiYWNrdXApIGUgYWNlc3NvOwplKSBEZWNsYXJhIHF1ZSBvIGRvY3VtZW50byBzdWJtZXRpZG8gw6kgbyBzZXUgdHJhYmFsaG8gb3JpZ2luYWwgZSBxdWUgZGV0w6ltIG8gZGlyZWl0byBkZSBjb25jZWRlciBhIHRlcmNlaXJvcyBvcyBkaXJlaXRvcyBjb250aWRvcyBuZXN0YSBsaWNlbsOnYS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBhIGVudHJlZ2EgZG8gZG9jdW1lbnRvIG7Do28gaW5mcmluZ2Ugb3MgZGlyZWl0b3MgZGUgb3V0cmEgcGVzc29hIG91IGVudGlkYWRlOwpmKSBEZWNsYXJhIHF1ZSwgbm8gY2FzbyBkbyBkb2N1bWVudG8gc3VibWV0aWRvIGNvbnRlciBtYXRlcmlhbCBkbyBxdWFsIG7Do28gZGV0w6ltIG9zIGRpcmVpdG9zIGRlCmF1dG9yLCBvYnRldmUgYSBhdXRvcml6YcOnw6NvIGlycmVzdHJpdGEgZG8gcmVzcGVjdGl2byBkZXRlbnRvciBkZXNzZXMgZGlyZWl0b3MgcGFyYSBjZWRlciDDoApVRlBFIG9zIGRpcmVpdG9zIHJlcXVlcmlkb3MgcG9yIGVzdGEgTGljZW7Dp2EgZSBhdXRvcml6YXIgYSB1bml2ZXJzaWRhZGUgYSB1dGlsaXrDoS1sb3MgbGVnYWxtZW50ZS4gRGVjbGFyYSB0YW1iw6ltIHF1ZSBlc3NlIG1hdGVyaWFsIGN1am9zIGRpcmVpdG9zIHPDo28gZGUgdGVyY2Vpcm9zIGVzdMOhIGNsYXJhbWVudGUgaWRlbnRpZmljYWRvIGUgcmVjb25oZWNpZG8gbm8gdGV4dG8gb3UgY29udGXDumRvIGRvIGRvY3VtZW50byBlbnRyZWd1ZTsKZykgU2UgbyBkb2N1bWVudG8gZW50cmVndWUgw6kgYmFzZWFkbyBlbSB0cmFiYWxobyBmaW5hbmNpYWRvIG91IGFwb2lhZG8gcG9yIG91dHJhIGluc3RpdHVpw6fDo28gcXVlIG7Do28gYSBVRlBFLCBkZWNsYXJhIHF1ZSBjdW1wcml1IHF1YWlzcXVlciBvYnJpZ2HDp8O1ZXMgZXhpZ2lkYXMgcGVsbyByZXNwZWN0aXZvIGNvbnRyYXRvIG91IGFjb3Jkby4KCkEgVUZQRSBpZGVudGlmaWNhcsOhIGNsYXJhbWVudGUgbyhzKSBub21lKHMpIGRvKHMpIGF1dG9yIChlcykgZG9zIGRpcmVpdG9zIGRvIGRvY3VtZW50byBlbnRyZWd1ZSBlIG7Do28gZmFyw6EgcXVhbHF1ZXIgYWx0ZXJhw6fDo28sIHBhcmEgYWzDqW0gZG8gcHJldmlzdG8gbmEgYWzDrW5lYSBjKS4KRepositório InstitucionalPUBhttps://repositorio.ufpe.br/oai/requestattena@ufpe.bropendoar:22212021-03-27T05:15:41Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE)false |
dc.title.pt_BR.fl_str_mv |
Squeezed very deep convolutional neural networks for text classification |
title |
Squeezed very deep convolutional neural networks for text classification |
spellingShingle |
Squeezed very deep convolutional neural networks for text classification SANTOS, Luã Lázaro Jesus dos Inteligência computacional Redes neurais convolucionais |
title_short |
Squeezed very deep convolutional neural networks for text classification |
title_full |
Squeezed very deep convolutional neural networks for text classification |
title_fullStr |
Squeezed very deep convolutional neural networks for text classification |
title_full_unstemmed |
Squeezed very deep convolutional neural networks for text classification |
title_sort |
Squeezed very deep convolutional neural networks for text classification |
author |
SANTOS, Luã Lázaro Jesus dos |
author_facet |
SANTOS, Luã Lázaro Jesus dos |
author_role |
author |
dc.contributor.authorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/7216467413729634 |
dc.contributor.advisorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/1244195230407619 |
dc.contributor.author.fl_str_mv |
SANTOS, Luã Lázaro Jesus dos |
dc.contributor.advisor1.fl_str_mv |
ZANCHETTIN, Cleber |
contributor_str_mv |
ZANCHETTIN, Cleber |
dc.subject.por.fl_str_mv |
Inteligência computacional Redes neurais convolucionais |
topic |
Inteligência computacional Redes neurais convolucionais |
description |
Embedding artificial intelligence on constrained platforms has become a trend since the growth of embedded systems and mobile devices, experimented in recent years. Al though constrained platforms do not have enough processing capabilities to train a sophis ticated deep learning model, like Convolutional Neural Network (CNN), they are already capable of performing inference locally by using a previously trained embedded model. This approach enables numerous advantages such as more privacy, smaller response la tency, and no real-time network dependence. Still, the use of a local CNN model on constrained platforms is restricted by its storage size and processing power. Most of the research in CNN has focused on increasing network depth to improve accuracy. In the text classification area, deep models were proposed with excellent performance but rely ing on large architectures with thousands of parameters, and consequently, they require high storage size and processing. One of the models with much renown is the Very Deep Convolutional Neural Networks (VDCNN). In this dissertation, it is proposed an archi tectural modification in the VDCNN model to reduce its storage size while keeping its performance. In this optimization process, the impacts of using Temporal Depthwise Sep arable Convolutions and Global Average Pooling in the network are evaluated regarding parameters, storage size, dedicated hardware dependence, and accuracy. The proposed Squeezed Very Deep Convolutional Neural Networks (SVDCNN) model is between 10x and 20x smaller than the original version, depending on the network depth, maintain ing a maximum disk size of 6MB. Regarding accuracy, the network experiences a loss between 0.1% and 1.0% in the accuracy performance while obtains lower latency over non-dedicated hardware and higher inference time ratio compared to the baseline model. |
publishDate |
2020 |
dc.date.issued.fl_str_mv |
2020-02-14 |
dc.date.accessioned.fl_str_mv |
2021-03-26T15:50:24Z |
dc.date.available.fl_str_mv |
2021-03-26T15:50:24Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.citation.fl_str_mv |
SANTOS, Luã Lázaro Jesus dos. Squeezed very deep convolutional neural networks for text classification. 2020. Dissertação (Mestrado em Ciência da Computação) - Universidade Federal de Pernambuco, Recife, 2020. |
dc.identifier.uri.fl_str_mv |
https://repositorio.ufpe.br/handle/123456789/39490 |
identifier_str_mv |
SANTOS, Luã Lázaro Jesus dos. Squeezed very deep convolutional neural networks for text classification. 2020. Dissertação (Mestrado em Ciência da Computação) - Universidade Federal de Pernambuco, Recife, 2020. |
url |
https://repositorio.ufpe.br/handle/123456789/39490 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
http://creativecommons.org/licenses/by-nc-nd/3.0/br/ info:eu-repo/semantics/openAccess |
rights_invalid_str_mv |
http://creativecommons.org/licenses/by-nc-nd/3.0/br/ |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.publisher.program.fl_str_mv |
Programa de Pos Graduacao em Ciencia da Computacao |
dc.publisher.initials.fl_str_mv |
UFPE |
dc.publisher.country.fl_str_mv |
Brasil |
publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFPE instname:Universidade Federal de Pernambuco (UFPE) instacron:UFPE |
instname_str |
Universidade Federal de Pernambuco (UFPE) |
instacron_str |
UFPE |
institution |
UFPE |
reponame_str |
Repositório Institucional da UFPE |
collection |
Repositório Institucional da UFPE |
bitstream.url.fl_str_mv |
https://repositorio.ufpe.br/bitstream/123456789/39490/1/DISSERTA%c3%87%c3%83O%20Lu%c3%a3%20L%c3%a1zaro%20Jesus%20dos%20Santos.pdf https://repositorio.ufpe.br/bitstream/123456789/39490/4/DISSERTA%c3%87%c3%83O%20Lu%c3%a3%20L%c3%a1zaro%20Jesus%20dos%20Santos.pdf.txt https://repositorio.ufpe.br/bitstream/123456789/39490/5/DISSERTA%c3%87%c3%83O%20Lu%c3%a3%20L%c3%a1zaro%20Jesus%20dos%20Santos.pdf.jpg https://repositorio.ufpe.br/bitstream/123456789/39490/2/license_rdf https://repositorio.ufpe.br/bitstream/123456789/39490/3/license.txt |
bitstream.checksum.fl_str_mv |
3434fa0bd6d5becdabb64182b4854997 80b183d7220a4ee84bd1bb804d6c1e89 9510b15fa1a8ab077d51ad1c211563ee e39d27027a6cc9cb039ad269a5db8e34 bd573a5ca8288eb7272482765f819534 |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE) |
repository.mail.fl_str_mv |
attena@ufpe.br |
_version_ |
1797780376531238912 |