MaxDropout: Deep neural network regularization based on maximum output values
Main Author: | |
---|---|
Publication Date: | 2020 |
Other Authors: | , , |
Format: | Conference object |
Language: | eng |
Source: | Repositório Institucional da UNESP |
Download full: | http://dx.doi.org/10.1109/ICPR48806.2021.9412733 http://hdl.handle.net/11449/233274 |
Summary: | Different techniques have emerged in the deep learning scenario, such as Convolutional Neural Networks, Deep Belief Networks, and Long Short-Term Memory Networks, to cite a few. In lockstep, regularization methods, which aim to prevent overfitting by penalizing the weight connections, or turning off some units, have been widely studied either. In this paper, we present a novel approach called MaxDropout, a regularizer for deep neural network models that works in a supervised fashion by removing (shutting off) the prominent neurons (i.e., most active) in each hidden layer. The model forces fewer activated units to learn more representative information, thus providing sparsity. Regarding the experiments, we show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout. The proposed method was evaluated in image classification, achieving comparable results to existing regularizers, such as Cutout and RandomErasing, also improving the accuracy of neural networks that uses Dropout by replacing the existing layer by MaxDropout. |
id |
UNSP_061025b564e2d3c228a9ad39f640f0c5 |
---|---|
oai_identifier_str |
oai:repositorio.unesp.br:11449/233274 |
network_acronym_str |
UNSP |
network_name_str |
Repositório Institucional da UNESP |
repository_id_str |
2946 |
spelling |
MaxDropout: Deep neural network regularization based on maximum output valuesDifferent techniques have emerged in the deep learning scenario, such as Convolutional Neural Networks, Deep Belief Networks, and Long Short-Term Memory Networks, to cite a few. In lockstep, regularization methods, which aim to prevent overfitting by penalizing the weight connections, or turning off some units, have been widely studied either. In this paper, we present a novel approach called MaxDropout, a regularizer for deep neural network models that works in a supervised fashion by removing (shutting off) the prominent neurons (i.e., most active) in each hidden layer. The model forces fewer activated units to learn more representative information, thus providing sparsity. Regarding the experiments, we show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout. The proposed method was evaluated in image classification, achieving comparable results to existing regularizers, such as Cutout and RandomErasing, also improving the accuracy of neural networks that uses Dropout by replacing the existing layer by MaxDropout.Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)Federal University of Sao Carlos - UFSCarPetróleo Brasileiro - PetrobrasSão Paulo State University - UNESPSão Paulo State University - UNESPFAPESP: #2013/07375-0FAPESP: #2014/12236-1FAPESP: #2017/25908-6FAPESP: #2019/07825-1CNPq: #307066/2017-7CNPq: #427968/2018-6Universidade Federal de São Carlos (UFSCar)Petróleo Brasileiro - PetrobrasUniversidade Estadual Paulista (UNESP)do Santos, Claudio Filipi GoncalvesColombo, DaniloRoder, Mateus [UNESP]Papa, João Paulo [UNESP]2022-05-01T06:02:36Z2022-05-01T06:02:36Z2020-01-01info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/conferenceObject2671-2676http://dx.doi.org/10.1109/ICPR48806.2021.9412733Proceedings - International Conference on Pattern Recognition, p. 2671-2676.1051-4651http://hdl.handle.net/11449/23327410.1109/ICPR48806.2021.94127332-s2.0-85110459046Scopusreponame:Repositório Institucional da UNESPinstname:Universidade Estadual Paulista (UNESP)instacron:UNESPengProceedings - International Conference on Pattern Recognitioninfo:eu-repo/semantics/openAccess2024-04-23T16:11:34Zoai:repositorio.unesp.br:11449/233274Repositório InstitucionalPUBhttp://repositorio.unesp.br/oai/requestopendoar:29462024-04-23T16:11:34Repositório Institucional da UNESP - Universidade Estadual Paulista (UNESP)false |
dc.title.none.fl_str_mv |
MaxDropout: Deep neural network regularization based on maximum output values |
title |
MaxDropout: Deep neural network regularization based on maximum output values |
spellingShingle |
MaxDropout: Deep neural network regularization based on maximum output values do Santos, Claudio Filipi Goncalves |
title_short |
MaxDropout: Deep neural network regularization based on maximum output values |
title_full |
MaxDropout: Deep neural network regularization based on maximum output values |
title_fullStr |
MaxDropout: Deep neural network regularization based on maximum output values |
title_full_unstemmed |
MaxDropout: Deep neural network regularization based on maximum output values |
title_sort |
MaxDropout: Deep neural network regularization based on maximum output values |
author |
do Santos, Claudio Filipi Goncalves |
author_facet |
do Santos, Claudio Filipi Goncalves Colombo, Danilo Roder, Mateus [UNESP] Papa, João Paulo [UNESP] |
author_role |
author |
author2 |
Colombo, Danilo Roder, Mateus [UNESP] Papa, João Paulo [UNESP] |
author2_role |
author author author |
dc.contributor.none.fl_str_mv |
Universidade Federal de São Carlos (UFSCar) Petróleo Brasileiro - Petrobras Universidade Estadual Paulista (UNESP) |
dc.contributor.author.fl_str_mv |
do Santos, Claudio Filipi Goncalves Colombo, Danilo Roder, Mateus [UNESP] Papa, João Paulo [UNESP] |
description |
Different techniques have emerged in the deep learning scenario, such as Convolutional Neural Networks, Deep Belief Networks, and Long Short-Term Memory Networks, to cite a few. In lockstep, regularization methods, which aim to prevent overfitting by penalizing the weight connections, or turning off some units, have been widely studied either. In this paper, we present a novel approach called MaxDropout, a regularizer for deep neural network models that works in a supervised fashion by removing (shutting off) the prominent neurons (i.e., most active) in each hidden layer. The model forces fewer activated units to learn more representative information, thus providing sparsity. Regarding the experiments, we show that it is possible to improve existing neural networks and provide better results in neural networks when Dropout is replaced by MaxDropout. The proposed method was evaluated in image classification, achieving comparable results to existing regularizers, such as Cutout and RandomErasing, also improving the accuracy of neural networks that uses Dropout by replacing the existing layer by MaxDropout. |
publishDate |
2020 |
dc.date.none.fl_str_mv |
2020-01-01 2022-05-01T06:02:36Z 2022-05-01T06:02:36Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/conferenceObject |
format |
conferenceObject |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://dx.doi.org/10.1109/ICPR48806.2021.9412733 Proceedings - International Conference on Pattern Recognition, p. 2671-2676. 1051-4651 http://hdl.handle.net/11449/233274 10.1109/ICPR48806.2021.9412733 2-s2.0-85110459046 |
url |
http://dx.doi.org/10.1109/ICPR48806.2021.9412733 http://hdl.handle.net/11449/233274 |
identifier_str_mv |
Proceedings - International Conference on Pattern Recognition, p. 2671-2676. 1051-4651 10.1109/ICPR48806.2021.9412733 2-s2.0-85110459046 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.relation.none.fl_str_mv |
Proceedings - International Conference on Pattern Recognition |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.format.none.fl_str_mv |
2671-2676 |
dc.source.none.fl_str_mv |
Scopus reponame:Repositório Institucional da UNESP instname:Universidade Estadual Paulista (UNESP) instacron:UNESP |
instname_str |
Universidade Estadual Paulista (UNESP) |
instacron_str |
UNESP |
institution |
UNESP |
reponame_str |
Repositório Institucional da UNESP |
collection |
Repositório Institucional da UNESP |
repository.name.fl_str_mv |
Repositório Institucional da UNESP - Universidade Estadual Paulista (UNESP) |
repository.mail.fl_str_mv |
|
_version_ |
1797790334715953152 |