Γ-IRT : an item response theory model for evaluating regression algorithms
Autor(a) principal: | |
---|---|
Data de Publicação: | 2021 |
Tipo de documento: | Dissertação |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UFPE |
Texto Completo: | https://repositorio.ufpe.br/handle/123456789/50976 |
Resumo: | Item Response Theory (IRT) is used to measure latent abilities of human respondents based on their responses to items with different difficulty levels. Recently, IRT has been applied to algorithm evaluation in Artificial Inteligence (AI), by treating the algorithms as respondents and the AI tasks as items. The most common models in IRT only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). Hence they are not adequate in application contexts where responses are recorded in a continuous scale. In this dissertation we propose the Γ-IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination. |
id |
UFPE_6a3311c2a4c635ef155e1f391fd7cc2e |
---|---|
oai_identifier_str |
oai:repositorio.ufpe.br:123456789/50976 |
network_acronym_str |
UFPE |
network_name_str |
Repositório Institucional da UFPE |
repository_id_str |
2221 |
spelling |
MORAES, João Victor Camposhttp://lattes.cnpq.br/6417754781077123http://lattes.cnpq.br/2984888073123287http://lattes.cnpq.br/4640945954423515PRUDÊNCIO, Ricardo Bastos CavalcanteSILVA FILHO, Telmo de Menezes e2023-06-12T12:58:28Z2023-06-12T12:58:28Z2021-03-09MORAES, João Victor Campos. Γ-IRT: an item response theory model for evaluating regression algorithms. 2021. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2021.https://repositorio.ufpe.br/handle/123456789/50976Item Response Theory (IRT) is used to measure latent abilities of human respondents based on their responses to items with different difficulty levels. Recently, IRT has been applied to algorithm evaluation in Artificial Inteligence (AI), by treating the algorithms as respondents and the AI tasks as items. The most common models in IRT only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). Hence they are not adequate in application contexts where responses are recorded in a continuous scale. In this dissertation we propose the Γ-IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination.FACEPETeoria da Resposta ao Item (IRT) é usada para medir habilidades latentes de respondentes humanos com base em suas respostas a itens com diferentes níveis de dificuldade. Recentemente, IRT tem sido aplicada à avaliação de algoritmos de Inteligência Artificial (IA), tratando os algoritmos como respondentes e as tarefas de IA como itens. Os modelos mais comuns em IRT lidam apenas com respostas dicotômicas (ou seja, uma resposta deve ser correta ou incorreta). Portanto, não são adequados em contextos de aplicação onde as respostas são registradas em escala contínua. Nesta dissertação propomos o modelo Γ-IRT, especialmente concebido para lidar com respostas positivas ilimitadas, que modelamos usando uma distribuição Gama, parametrizada de acordo com a habilidade do respondente e parâmetros de dificuldade e discriminação do item. A parametrização proposta resulta em curvas características de itens com formatos mais flexíveis em relação às curvas logísticas tradicionais adotadas em IRT. Aplicamos o modelo proposto para avaliar as habilidades do modelo de regressão, onde as respostas são os erros absolutos nas instâncias de teste. Esta nova aplicação representa uma alternativa para avaliar o desempenho da regressão e para identificar regiões em um conjunto de dados de regressão que apresentam diferentes níveis de dificuldade e discriminação.engUniversidade Federal de PernambucoPrograma de Pos Graduacao em Ciencia da ComputacaoUFPEBrasilhttp://creativecommons.org/licenses/by-nc-nd/3.0/br/info:eu-repo/semantics/openAccessInteligência artificialAprendizagem de máquinaΓ-IRT : an item response theory model for evaluating regression algorithmsinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesismestradoreponame:Repositório Institucional da UFPEinstname:Universidade Federal de Pernambuco (UFPE)instacron:UFPELICENSElicense.txtlicense.txttext/plain; charset=utf-82362https://repositorio.ufpe.br/bitstream/123456789/50976/3/license.txt5e89a1613ddc8510c6576f4b23a78973MD53ORIGINALDISSERTAÇÃO João Victor Campos Moraes.pdfDISSERTAÇÃO João Victor Campos Moraes.pdfapplication/pdf2832007https://repositorio.ufpe.br/bitstream/123456789/50976/1/DISSERTA%c3%87%c3%83O%20Jo%c3%a3o%20Victor%20Campos%20Moraes.pdfcee4eb50fde0f340625b34d262df31c4MD51CC-LICENSElicense_rdflicense_rdfapplication/rdf+xml; charset=utf-8811https://repositorio.ufpe.br/bitstream/123456789/50976/2/license_rdfe39d27027a6cc9cb039ad269a5db8e34MD52TEXTDISSERTAÇÃO João Victor Campos Moraes.pdf.txtDISSERTAÇÃO João Victor Campos Moraes.pdf.txtExtracted texttext/plain88625https://repositorio.ufpe.br/bitstream/123456789/50976/4/DISSERTA%c3%87%c3%83O%20Jo%c3%a3o%20Victor%20Campos%20Moraes.pdf.txt92ae70292ace5dd928652f9ed615dd31MD54THUMBNAILDISSERTAÇÃO João Victor Campos Moraes.pdf.jpgDISSERTAÇÃO João Victor Campos Moraes.pdf.jpgGenerated Thumbnailimage/jpeg1274https://repositorio.ufpe.br/bitstream/123456789/50976/5/DISSERTA%c3%87%c3%83O%20Jo%c3%a3o%20Victor%20Campos%20Moraes.pdf.jpg9f7e18db5a71caaf093f4f8754aab02cMD55123456789/509762023-06-13 02:34:02.21oai:repositorio.ufpe.br:123456789/50976VGVybW8gZGUgRGVww7NzaXRvIExlZ2FsIGUgQXV0b3JpemHDp8OjbyBwYXJhIFB1YmxpY2l6YcOnw6NvIGRlIERvY3VtZW50b3Mgbm8gUmVwb3NpdMOzcmlvIERpZ2l0YWwgZGEgVUZQRQoKCkRlY2xhcm8gZXN0YXIgY2llbnRlIGRlIHF1ZSBlc3RlIFRlcm1vIGRlIERlcMOzc2l0byBMZWdhbCBlIEF1dG9yaXphw6fDo28gdGVtIG8gb2JqZXRpdm8gZGUgZGl2dWxnYcOnw6NvIGRvcyBkb2N1bWVudG9zIGRlcG9zaXRhZG9zIG5vIFJlcG9zaXTDs3JpbyBEaWdpdGFsIGRhIFVGUEUgZSBkZWNsYXJvIHF1ZToKCkkgLSBvcyBkYWRvcyBwcmVlbmNoaWRvcyBubyBmb3JtdWzDoXJpbyBkZSBkZXDDs3NpdG8gc8OjbyB2ZXJkYWRlaXJvcyBlIGF1dMOqbnRpY29zOwoKSUkgLSAgbyBjb250ZcO6ZG8gZGlzcG9uaWJpbGl6YWRvIMOpIGRlIHJlc3BvbnNhYmlsaWRhZGUgZGUgc3VhIGF1dG9yaWE7CgpJSUkgLSBvIGNvbnRlw7pkbyDDqSBvcmlnaW5hbCwgZSBzZSBvIHRyYWJhbGhvIGUvb3UgcGFsYXZyYXMgZGUgb3V0cmFzIHBlc3NvYXMgZm9yYW0gdXRpbGl6YWRvcywgZXN0YXMgZm9yYW0gZGV2aWRhbWVudGUgcmVjb25oZWNpZGFzOwoKSVYgLSBxdWFuZG8gdHJhdGFyLXNlIGRlIG9icmEgY29sZXRpdmEgKG1haXMgZGUgdW0gYXV0b3IpOiB0b2RvcyBvcyBhdXRvcmVzIGVzdMOjbyBjaWVudGVzIGRvIGRlcMOzc2l0byBlIGRlIGFjb3JkbyBjb20gZXN0ZSB0ZXJtbzsKClYgLSBxdWFuZG8gdHJhdGFyLXNlIGRlIFRyYWJhbGhvIGRlIENvbmNsdXPDo28gZGUgQ3Vyc28sIERpc3NlcnRhw6fDo28gb3UgVGVzZTogbyBhcnF1aXZvIGRlcG9zaXRhZG8gY29ycmVzcG9uZGUgw6AgdmVyc8OjbyBmaW5hbCBkbyB0cmFiYWxobzsKClZJIC0gcXVhbmRvIHRyYXRhci1zZSBkZSBUcmFiYWxobyBkZSBDb25jbHVzw6NvIGRlIEN1cnNvLCBEaXNzZXJ0YcOnw6NvIG91IFRlc2U6IGVzdG91IGNpZW50ZSBkZSBxdWUgYSBhbHRlcmHDp8OjbyBkYSBtb2RhbGlkYWRlIGRlIGFjZXNzbyBhbyBkb2N1bWVudG8gYXDDs3MgbyBkZXDDs3NpdG8gZSBhbnRlcyBkZSBmaW5kYXIgbyBwZXLDrW9kbyBkZSBlbWJhcmdvLCBxdWFuZG8gZm9yIGVzY29saGlkbyBhY2Vzc28gcmVzdHJpdG8sIHNlcsOhIHBlcm1pdGlkYSBtZWRpYW50ZSBzb2xpY2l0YcOnw6NvIGRvIChhKSBhdXRvciAoYSkgYW8gU2lzdGVtYSBJbnRlZ3JhZG8gZGUgQmlibGlvdGVjYXMgZGEgVUZQRSAoU0lCL1VGUEUpLgoKIApQYXJhIHRyYWJhbGhvcyBlbSBBY2Vzc28gQWJlcnRvOgoKTmEgcXVhbGlkYWRlIGRlIHRpdHVsYXIgZG9zIGRpcmVpdG9zIGF1dG9yYWlzIGRlIGF1dG9yIHF1ZSByZWNhZW0gc29icmUgZXN0ZSBkb2N1bWVudG8sIGZ1bmRhbWVudGFkbyBuYSBMZWkgZGUgRGlyZWl0byBBdXRvcmFsIG5vIDkuNjEwLCBkZSAxOSBkZSBmZXZlcmVpcm8gZGUgMTk5OCwgYXJ0LiAyOSwgaW5jaXNvIElJSSwgYXV0b3Jpem8gYSBVbml2ZXJzaWRhZGUgRmVkZXJhbCBkZSBQZXJuYW1idWNvIGEgZGlzcG9uaWJpbGl6YXIgZ3JhdHVpdGFtZW50ZSwgc2VtIHJlc3NhcmNpbWVudG8gZG9zIGRpcmVpdG9zIGF1dG9yYWlzLCBwYXJhIGZpbnMgZGUgbGVpdHVyYSwgaW1wcmVzc8OjbyBlL291IGRvd25sb2FkIChhcXVpc2nDp8OjbykgYXRyYXbDqXMgZG8gc2l0ZSBkbyBSZXBvc2l0w7NyaW8gRGlnaXRhbCBkYSBVRlBFIG5vIGVuZGVyZcOnbyBodHRwOi8vd3d3LnJlcG9zaXRvcmlvLnVmcGUuYnIsIGEgcGFydGlyIGRhIGRhdGEgZGUgZGVww7NzaXRvLgoKIApQYXJhIHRyYWJhbGhvcyBlbSBBY2Vzc28gUmVzdHJpdG86CgpOYSBxdWFsaWRhZGUgZGUgdGl0dWxhciBkb3MgZGlyZWl0b3MgYXV0b3JhaXMgZGUgYXV0b3IgcXVlIHJlY2FlbSBzb2JyZSBlc3RlIGRvY3VtZW50bywgZnVuZGFtZW50YWRvIG5hIExlaSBkZSBEaXJlaXRvIEF1dG9yYWwgbm8gOS42MTAgZGUgMTkgZGUgZmV2ZXJlaXJvIGRlIDE5OTgsIGFydC4gMjksIGluY2lzbyBJSUksIGF1dG9yaXpvIGEgVW5pdmVyc2lkYWRlIEZlZGVyYWwgZGUgUGVybmFtYnVjbyBhIGRpc3BvbmliaWxpemFyIGdyYXR1aXRhbWVudGUsIHNlbSByZXNzYXJjaW1lbnRvIGRvcyBkaXJlaXRvcyBhdXRvcmFpcywgcGFyYSBmaW5zIGRlIGxlaXR1cmEsIGltcHJlc3PDo28gZS9vdSBkb3dubG9hZCAoYXF1aXNpw6fDo28pIGF0cmF2w6lzIGRvIHNpdGUgZG8gUmVwb3NpdMOzcmlvIERpZ2l0YWwgZGEgVUZQRSBubyBlbmRlcmXDp28gaHR0cDovL3d3dy5yZXBvc2l0b3Jpby51ZnBlLmJyLCBxdWFuZG8gZmluZGFyIG8gcGVyw61vZG8gZGUgZW1iYXJnbyBjb25kaXplbnRlIGFvIHRpcG8gZGUgZG9jdW1lbnRvLCBjb25mb3JtZSBpbmRpY2FkbyBubyBjYW1wbyBEYXRhIGRlIEVtYmFyZ28uCg==Repositório InstitucionalPUBhttps://repositorio.ufpe.br/oai/requestattena@ufpe.bropendoar:22212023-06-13T05:34:02Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE)false |
dc.title.pt_BR.fl_str_mv |
Γ-IRT : an item response theory model for evaluating regression algorithms |
title |
Γ-IRT : an item response theory model for evaluating regression algorithms |
spellingShingle |
Γ-IRT : an item response theory model for evaluating regression algorithms MORAES, João Victor Campos Inteligência artificial Aprendizagem de máquina |
title_short |
Γ-IRT : an item response theory model for evaluating regression algorithms |
title_full |
Γ-IRT : an item response theory model for evaluating regression algorithms |
title_fullStr |
Γ-IRT : an item response theory model for evaluating regression algorithms |
title_full_unstemmed |
Γ-IRT : an item response theory model for evaluating regression algorithms |
title_sort |
Γ-IRT : an item response theory model for evaluating regression algorithms |
author |
MORAES, João Victor Campos |
author_facet |
MORAES, João Victor Campos |
author_role |
author |
dc.contributor.authorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/6417754781077123 |
dc.contributor.advisorLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/2984888073123287 |
dc.contributor.advisor-coLattes.pt_BR.fl_str_mv |
http://lattes.cnpq.br/4640945954423515 |
dc.contributor.author.fl_str_mv |
MORAES, João Victor Campos |
dc.contributor.advisor1.fl_str_mv |
PRUDÊNCIO, Ricardo Bastos Cavalcante |
dc.contributor.advisor-co1.fl_str_mv |
SILVA FILHO, Telmo de Menezes e |
contributor_str_mv |
PRUDÊNCIO, Ricardo Bastos Cavalcante SILVA FILHO, Telmo de Menezes e |
dc.subject.por.fl_str_mv |
Inteligência artificial Aprendizagem de máquina |
topic |
Inteligência artificial Aprendizagem de máquina |
description |
Item Response Theory (IRT) is used to measure latent abilities of human respondents based on their responses to items with different difficulty levels. Recently, IRT has been applied to algorithm evaluation in Artificial Inteligence (AI), by treating the algorithms as respondents and the AI tasks as items. The most common models in IRT only deal with dichotomous responses (i.e., a response has to be either correct or incorrect). Hence they are not adequate in application contexts where responses are recorded in a continuous scale. In this dissertation we propose the Γ-IRT model, particularly designed for dealing with positive unbounded responses, which we model using a Gamma distribution, parameterised according to respondent ability and item difficulty and discrimination parameters. The proposed parameterisation results in item characteristic curves with more flexible shapes compared to the traditional logistic curves adopted in IRT. We apply the proposed model to assess regression model abilities, where responses are the absolute errors in test instances. This novel application represents an alternative for evaluating regression performance and for identifying regions in a regression dataset that present different levels of difficulty and discrimination. |
publishDate |
2021 |
dc.date.issued.fl_str_mv |
2021-03-09 |
dc.date.accessioned.fl_str_mv |
2023-06-12T12:58:28Z |
dc.date.available.fl_str_mv |
2023-06-12T12:58:28Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/masterThesis |
format |
masterThesis |
status_str |
publishedVersion |
dc.identifier.citation.fl_str_mv |
MORAES, João Victor Campos. Γ-IRT: an item response theory model for evaluating regression algorithms. 2021. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2021. |
dc.identifier.uri.fl_str_mv |
https://repositorio.ufpe.br/handle/123456789/50976 |
identifier_str_mv |
MORAES, João Victor Campos. Γ-IRT: an item response theory model for evaluating regression algorithms. 2021. Dissertação (Mestrado em Ciência da Computação) – Universidade Federal de Pernambuco, Recife, 2021. |
url |
https://repositorio.ufpe.br/handle/123456789/50976 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
http://creativecommons.org/licenses/by-nc-nd/3.0/br/ info:eu-repo/semantics/openAccess |
rights_invalid_str_mv |
http://creativecommons.org/licenses/by-nc-nd/3.0/br/ |
eu_rights_str_mv |
openAccess |
dc.publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.publisher.program.fl_str_mv |
Programa de Pos Graduacao em Ciencia da Computacao |
dc.publisher.initials.fl_str_mv |
UFPE |
dc.publisher.country.fl_str_mv |
Brasil |
publisher.none.fl_str_mv |
Universidade Federal de Pernambuco |
dc.source.none.fl_str_mv |
reponame:Repositório Institucional da UFPE instname:Universidade Federal de Pernambuco (UFPE) instacron:UFPE |
instname_str |
Universidade Federal de Pernambuco (UFPE) |
instacron_str |
UFPE |
institution |
UFPE |
reponame_str |
Repositório Institucional da UFPE |
collection |
Repositório Institucional da UFPE |
bitstream.url.fl_str_mv |
https://repositorio.ufpe.br/bitstream/123456789/50976/3/license.txt https://repositorio.ufpe.br/bitstream/123456789/50976/1/DISSERTA%c3%87%c3%83O%20Jo%c3%a3o%20Victor%20Campos%20Moraes.pdf https://repositorio.ufpe.br/bitstream/123456789/50976/2/license_rdf https://repositorio.ufpe.br/bitstream/123456789/50976/4/DISSERTA%c3%87%c3%83O%20Jo%c3%a3o%20Victor%20Campos%20Moraes.pdf.txt https://repositorio.ufpe.br/bitstream/123456789/50976/5/DISSERTA%c3%87%c3%83O%20Jo%c3%a3o%20Victor%20Campos%20Moraes.pdf.jpg |
bitstream.checksum.fl_str_mv |
5e89a1613ddc8510c6576f4b23a78973 cee4eb50fde0f340625b34d262df31c4 e39d27027a6cc9cb039ad269a5db8e34 92ae70292ace5dd928652f9ed615dd31 9f7e18db5a71caaf093f4f8754aab02c |
bitstream.checksumAlgorithm.fl_str_mv |
MD5 MD5 MD5 MD5 MD5 |
repository.name.fl_str_mv |
Repositório Institucional da UFPE - Universidade Federal de Pernambuco (UFPE) |
repository.mail.fl_str_mv |
attena@ufpe.br |
_version_ |
1802310655385010176 |