Sample size estimation for power and accuracy in the experimental comparison of metaheuristics
Autor(a) principal: | |
---|---|
Data de Publicação: | 2018 |
Tipo de documento: | Tese |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UFMG |
Texto Completo: | http://hdl.handle.net/1843/32571 |
Resumo: | Experimental algorithmics encompasses the study of guidelines and methods for computational evaluation of algorithms. In the optimization field, it is useful for testing the performance of algorithms when solving a certain type of problem. In this work we develop a methodology for generating adequate experimental designs for comparing the performance of optimization metaheuristics, with a focus on statistical power and accuracy in parameter estimation. In particular, we deal with sample size estimation for experiments involving optimization algorithms, both in terms of within-instance repeated executions and the number of instances required. A statistically sound methodology is presented for sample size calculation, allowing relevant comparisons between the performances of two algorithms for a given class of problems. The methodology’s effectiveness is validated using simulated models and exemplified with two case studies. The proposed methodology was implemented in the form of an open source R package, published in the CRAN repository. |
id |
UFMG_0d2f545fbb27960dd116df3ed311cc6c |
---|---|
oai_identifier_str |
oai:repositorio.ufmg.br:1843/32571 |
network_acronym_str |
UFMG |
network_name_str |
Repositório Institucional da UFMG |
repository_id_str |
|
spelling |
Felipe Campelo Franca Pintohttp://lattes.cnpq.br/6799982843395323Claus de Castro AranhaThiago Ferreira de NoronhaLuiz Henrique DuczmalHélio José Corrêa Barbosahttp://lattes.cnpq.br/7262153522013649Fernanda Caldeira Takahashi2020-02-18T16:37:49Z2020-02-18T16:37:49Z2018-12-03http://hdl.handle.net/1843/32571Experimental algorithmics encompasses the study of guidelines and methods for computational evaluation of algorithms. In the optimization field, it is useful for testing the performance of algorithms when solving a certain type of problem. In this work we develop a methodology for generating adequate experimental designs for comparing the performance of optimization metaheuristics, with a focus on statistical power and accuracy in parameter estimation. In particular, we deal with sample size estimation for experiments involving optimization algorithms, both in terms of within-instance repeated executions and the number of instances required. A statistically sound methodology is presented for sample size calculation, allowing relevant comparisons between the performances of two algorithms for a given class of problems. The methodology’s effectiveness is validated using simulated models and exemplified with two case studies. The proposed methodology was implemented in the form of an open source R package, published in the CRAN repository.Experimentação algoritmica contempla o estudo de diretrizes e métodos para avaliação computacional de algoritmos. No campo da otimização, ela é útil para testar o desempenho de algoritmos ao resolver classes específicas de problemas. Nesse trabalho estamos desenvolvendo uma metodologia para geração planejamentos experimentais adequados para comparação de desempenhodemeta-heurísticas,comumfocoempotênciaestatísticaeprecisãonaestimação de parâmetros. Em particular, lidamos com estimação do tamanho amostral para experimentos que envolvem algoritmos de otimização, tanto em termos do número de execuções em uma mesma instância quanto do número de instâncias necessárias. Uma metodologia estatisticamente válida é apresentada para o calculo de tamanho amostral, permitindo comparações relevantes entre as performances de dois algoritmos para uma dada classe de problemas. A eficácia da metodologia é validada usando modelos simulados e exemplificada com dois estudos de caso. A metodologia proposta foi implementada na forma de pacote em R código aberto, publicado no repositório CRAN.engUniversidade Federal de Minas GeraisPrograma de Pós-Graduação em Engenharia ElétricaUFMGBrasilENG - DEPARTAMENTO DE ENGENHARIA ELÉTRICAEngenharia elétricaProgramação heurísticaAlgoritmosEngenharia elétricaSample size estimation for power and accuracy in the experimental comparison of metaheuristicsinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/doctoralThesisinfo:eu-repo/semantics/openAccessreponame:Repositório Institucional da UFMGinstname:Universidade Federal de Minas Gerais (UFMG)instacron:UFMGORIGINALFernandaCaldeiraTakahashi_TeseDOUTORADO.pdfFernandaCaldeiraTakahashi_TeseDOUTORADO.pdfapplication/pdf2182689https://repositorio.ufmg.br/bitstream/1843/32571/1/FernandaCaldeiraTakahashi_TeseDOUTORADO.pdfc20324e8ef5ec4eeddbd3d3814ea5defMD51LICENSElicense.txtlicense.txttext/plain; charset=utf-82119https://repositorio.ufmg.br/bitstream/1843/32571/2/license.txt34badce4be7e31e3adb4575ae96af679MD52TEXTFernandaCaldeiraTakahashi_TeseDOUTORADO.pdf.txtFernandaCaldeiraTakahashi_TeseDOUTORADO.pdf.txtExtracted texttext/plain177932https://repositorio.ufmg.br/bitstream/1843/32571/3/FernandaCaldeiraTakahashi_TeseDOUTORADO.pdf.txta620067f1c2669acbecc295914c32839MD531843/325712020-02-19 03:31:05.206oai:repositorio.ufmg.br:1843/32571TElDRU7Dh0EgREUgRElTVFJJQlVJw4fDg08gTsODTy1FWENMVVNJVkEgRE8gUkVQT1NJVMOTUklPIElOU1RJVFVDSU9OQUwgREEgVUZNRwoKQ29tIGEgYXByZXNlbnRhw6fDo28gZGVzdGEgbGljZW7Dp2EsIHZvY8OqIChvIGF1dG9yIChlcykgb3UgbyB0aXR1bGFyIGRvcyBkaXJlaXRvcyBkZSBhdXRvcikgY29uY2VkZSBhbyBSZXBvc2l0w7NyaW8gSW5zdGl0dWNpb25hbCBkYSBVRk1HIChSSS1VRk1HKSBvIGRpcmVpdG8gbsOjbyBleGNsdXNpdm8gZSBpcnJldm9nw6F2ZWwgZGUgcmVwcm9kdXppciBlL291IGRpc3RyaWJ1aXIgYSBzdWEgcHVibGljYcOnw6NvIChpbmNsdWluZG8gbyByZXN1bW8pIHBvciB0b2RvIG8gbXVuZG8gbm8gZm9ybWF0byBpbXByZXNzbyBlIGVsZXRyw7RuaWNvIGUgZW0gcXVhbHF1ZXIgbWVpbywgaW5jbHVpbmRvIG9zIGZvcm1hdG9zIMOhdWRpbyBvdSB2w61kZW8uCgpWb2PDqiBkZWNsYXJhIHF1ZSBjb25oZWNlIGEgcG9sw610aWNhIGRlIGNvcHlyaWdodCBkYSBlZGl0b3JhIGRvIHNldSBkb2N1bWVudG8gZSBxdWUgY29uaGVjZSBlIGFjZWl0YSBhcyBEaXJldHJpemVzIGRvIFJJLVVGTUcuCgpWb2PDqiBjb25jb3JkYSBxdWUgbyBSZXBvc2l0w7NyaW8gSW5zdGl0dWNpb25hbCBkYSBVRk1HIHBvZGUsIHNlbSBhbHRlcmFyIG8gY29udGXDumRvLCB0cmFuc3BvciBhIHN1YSBwdWJsaWNhw6fDo28gcGFyYSBxdWFscXVlciBtZWlvIG91IGZvcm1hdG8gcGFyYSBmaW5zIGRlIHByZXNlcnZhw6fDo28uCgpWb2PDqiB0YW1iw6ltIGNvbmNvcmRhIHF1ZSBvIFJlcG9zaXTDs3JpbyBJbnN0aXR1Y2lvbmFsIGRhIFVGTUcgcG9kZSBtYW50ZXIgbWFpcyBkZSB1bWEgY8OzcGlhIGRlIHN1YSBwdWJsaWNhw6fDo28gcGFyYSBmaW5zIGRlIHNlZ3VyYW7Dp2EsIGJhY2stdXAgZSBwcmVzZXJ2YcOnw6NvLgoKVm9jw6ogZGVjbGFyYSBxdWUgYSBzdWEgcHVibGljYcOnw6NvIMOpIG9yaWdpbmFsIGUgcXVlIHZvY8OqIHRlbSBvIHBvZGVyIGRlIGNvbmNlZGVyIG9zIGRpcmVpdG9zIGNvbnRpZG9zIG5lc3RhIGxpY2Vuw6dhLiBWb2PDqiB0YW1iw6ltIGRlY2xhcmEgcXVlIG8gZGVww7NzaXRvIGRlIHN1YSBwdWJsaWNhw6fDo28gbsOjbywgcXVlIHNlamEgZGUgc2V1IGNvbmhlY2ltZW50bywgaW5mcmluZ2UgZGlyZWl0b3MgYXV0b3JhaXMgZGUgbmluZ3XDqW0uCgpDYXNvIGEgc3VhIHB1YmxpY2HDp8OjbyBjb250ZW5oYSBtYXRlcmlhbCBxdWUgdm9jw6ogbsOjbyBwb3NzdWkgYSB0aXR1bGFyaWRhZGUgZG9zIGRpcmVpdG9zIGF1dG9yYWlzLCB2b2PDqiBkZWNsYXJhIHF1ZSBvYnRldmUgYSBwZXJtaXNzw6NvIGlycmVzdHJpdGEgZG8gZGV0ZW50b3IgZG9zIGRpcmVpdG9zIGF1dG9yYWlzIHBhcmEgY29uY2VkZXIgYW8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgZGEgVUZNRyBvcyBkaXJlaXRvcyBhcHJlc2VudGFkb3MgbmVzdGEgbGljZW7Dp2EsIGUgcXVlIGVzc2UgbWF0ZXJpYWwgZGUgcHJvcHJpZWRhZGUgZGUgdGVyY2Vpcm9zIGVzdMOhIGNsYXJhbWVudGUgaWRlbnRpZmljYWRvIGUgcmVjb25oZWNpZG8gbm8gdGV4dG8gb3Ugbm8gY29udGXDumRvIGRhIHB1YmxpY2HDp8OjbyBvcmEgZGVwb3NpdGFkYS4KCkNBU08gQSBQVUJMSUNBw4fDg08gT1JBIERFUE9TSVRBREEgVEVOSEEgU0lETyBSRVNVTFRBRE8gREUgVU0gUEFUUk9Dw41OSU8gT1UgQVBPSU8gREUgVU1BIEFHw4pOQ0lBIERFIEZPTUVOVE8gT1UgT1VUUk8gT1JHQU5JU01PLCBWT0PDiiBERUNMQVJBIFFVRSBSRVNQRUlUT1UgVE9ET1MgRSBRVUFJU1FVRVIgRElSRUlUT1MgREUgUkVWSVPDg08gQ09NTyBUQU1Cw4lNIEFTIERFTUFJUyBPQlJJR0HDh8OVRVMgRVhJR0lEQVMgUE9SIENPTlRSQVRPIE9VIEFDT1JETy4KCk8gUmVwb3NpdMOzcmlvIEluc3RpdHVjaW9uYWwgZGEgVUZNRyBzZSBjb21wcm9tZXRlIGEgaWRlbnRpZmljYXIgY2xhcmFtZW50ZSBvIHNldSBub21lKHMpIG91IG8ocykgbm9tZXMocykgZG8ocykgZGV0ZW50b3IoZXMpIGRvcyBkaXJlaXRvcyBhdXRvcmFpcyBkYSBwdWJsaWNhw6fDo28sIGUgbsOjbyBmYXLDoSBxdWFscXVlciBhbHRlcmHDp8OjbywgYWzDqW0gZGFxdWVsYXMgY29uY2VkaWRhcyBwb3IgZXN0YSBsaWNlbsOnYS4KCg==Repositório de PublicaçõesPUBhttps://repositorio.ufmg.br/oaiopendoar:2020-02-19T06:31:05Repositório Institucional da UFMG - Universidade Federal de Minas Gerais (UFMG)false |
dc.title.pt_BR.fl_str_mv |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
title |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
spellingShingle |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics Fernanda Caldeira Takahashi Engenharia elétrica Engenharia elétrica Programação heurística Algoritmos |
title_short |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
title_full |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
title_fullStr |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
title_full_unstemmed |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
title_sort |
Sample size estimation for power and accuracy in the experimental comparison of metaheuristics |
author |
Fernanda Caldeira Takahashi |
author_facet |
Fernanda Caldeira Takahashi |
author_role |
author |
dc.contributor.advisor1.fl_str_mv |
Felipe Campelo Franca Pinto |
dc.contributor.advisor1Lattes.fl_str_mv |
http://lattes.cnpq.br/6799982843395323 |
dc.contributor.referee1.fl_str_mv |
Claus de Castro Aranha |
dc.contributor.referee2.fl_str_mv |
Thiago Ferreira de Noronha |
dc.contributor.referee3.fl_str_mv |
Luiz Henrique Duczmal |
dc.contributor.referee4.fl_str_mv |
Hélio José Corrêa Barbosa |
dc.contributor.authorLattes.fl_str_mv |
http://lattes.cnpq.br/7262153522013649 |
dc.contributor.author.fl_str_mv |
Fernanda Caldeira Takahashi |
contributor_str_mv |
Felipe Campelo Franca Pinto Claus de Castro Aranha Thiago Ferreira de Noronha Luiz Henrique Duczmal Hélio José Corrêa Barbosa |
dc.subject.por.fl_str_mv |
Engenharia elétrica |
topic |
Engenharia elétrica Engenharia elétrica Programação heurística Algoritmos |
dc.subject.other.pt_BR.fl_str_mv |
Engenharia elétrica Programação heurística Algoritmos |
description |
Experimental algorithmics encompasses the study of guidelines and methods for computational evaluation of algorithms. In the optimization field, it is useful for testing the performance of algorithms when solving a certain type of problem. In this work we develop a methodology for generating adequate experimental designs for comparing the performance of optimization metaheuristics, with a focus on statistical power and accuracy in parameter estimation. In particular, we deal with sample size estimation for experiments involving optimization algorithms, both in terms of within-instance repeated executions and the number of instances required. A statistically sound methodology is presented for sample size calculation, allowing relevant comparisons between the performances of two algorithms for a given class of problems. The methodology’s effectiveness is validated using simulated models and exemplified with two case studies. The proposed methodology was implemented in the form of an open source R package, published in the CRAN repository. |
publishDate |
2018 |
dc.date.issued.fl_str_mv |
2018-12-03 |
dc.date.accessioned.fl_str_mv |
2020-02-18T16:37:49Z |
dc.date.available.fl_str_mv |
2020-02-18T16:37:49Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/doctoralThesis |
format |
doctoralThesis |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://hdl.handle.net/1843/32571 |
url |
http://hdl.handle.net/1843/32571 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Minas Gerais |
dc.publisher.program.fl_str_mv |
Programa de Pós-Graduação em Engenharia Elétrica |
dc.publisher.initials.fl_str_mv |
UFMG |
dc.publisher.country.fl_str_mv |
Brasil |
dc.publisher.department.fl_str_mv |
ENG - DEPARTAMENTO DE ENGENHARIA ELÉTRICA |
publisher.none.fl_str_mv |
Universidade Federal de Minas Gerais |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFMG instname:Universidade Federal de Minas Gerais (UFMG) instacron:UFMG |
instname_str |
Universidade Federal de Minas Gerais (UFMG) |
instacron_str |
UFMG |
institution |
UFMG |
reponame_str |
Repositório Institucional da UFMG |
collection |
Repositório Institucional da UFMG |
bitstream.url.fl_str_mv |
https://repositorio.ufmg.br/bitstream/1843/32571/1/FernandaCaldeiraTakahashi_TeseDOUTORADO.pdf https://repositorio.ufmg.br/bitstream/1843/32571/2/license.txt https://repositorio.ufmg.br/bitstream/1843/32571/3/FernandaCaldeiraTakahashi_TeseDOUTORADO.pdf.txt |
bitstream.checksum.fl_str_mv |
c20324e8ef5ec4eeddbd3d3814ea5def 34badce4be7e31e3adb4575ae96af679 a620067f1c2669acbecc295914c32839 |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFMG - Universidade Federal de Minas Gerais (UFMG) |
repository.mail.fl_str_mv |
|
_version_ |
1803589348368580608 |