Adaptations to the Heuristic Evaluation (HE) method for novice evaluators

Detalhes bibliográficos
Autor(a) principal: Salgado, André de Lima
Data de Publicação: 2017
Tipo de documento: Dissertação
Idioma: eng
Título da fonte: Biblioteca Digital de Teses e Dissertações da USP
Texto Completo: http://www.teses.usp.br/teses/disponiveis/55/55134/tde-08022018-084022/
Resumo: Heuristic Evaluation (HE) is a popular method of usability inspection. However, its outcomes are dependent on the expertise of evaluators. This study explored and described the difference in quality of outcomes (reports) of a collaborative HE conducted by evaluator groups of distinct composition, regarding different numbers of expert evaluators in each group. Twenty-seven (27) evaluators voluntarily contributed with this study, nine (9) expert and 18 novice evaluators. Thus, I organized seven (7) HE groups according to four (4) different levels of the factor presence of an expert, which ranged from no expert up to three (3) experts in the same group. Each group agreed to provide their reports for this study. Thereafter, I conducted a comparative analysis on the reports based on standard methods of the field and on a cluster analysis of similarities. I described the F-measure for each group report according to a relaxed and a strict criteria. Also, I described the dendrograms formed from the cluster analysis and the respective similarities indicated by each cluster. The results showed that the quality of reports from collaborative HE conducted by experts and novices together can be more similar to the quality of reports from a traditional HE with multiple expert inspectors (Benchmark Group) then to the quality of reports from a collaborative HE conducted by a group composed only by novice evaluators (Baseline Group). Finally, I discuss additional findings and implications for future studies in the field.
id USP_cbfdf7b74c81292ba18fdcdb1ceacb73
oai_identifier_str oai:teses.usp.br:tde-08022018-084022
network_acronym_str USP
network_name_str Biblioteca Digital de Teses e Dissertações da USP
repository_id_str 2721
spelling Adaptations to the Heuristic Evaluation (HE) method for novice evaluatorsAdaptações ao método de Avaliação Heurística (AH) para avaliadores novatosAvaliação heurísticaAvaliadores novatosEfeito avaliadorEfeito experiênciaEvaluator effectExpertise effectHeuristic evaluationNovice evaluatorsUsabilidadeUsabilityHeuristic Evaluation (HE) is a popular method of usability inspection. However, its outcomes are dependent on the expertise of evaluators. This study explored and described the difference in quality of outcomes (reports) of a collaborative HE conducted by evaluator groups of distinct composition, regarding different numbers of expert evaluators in each group. Twenty-seven (27) evaluators voluntarily contributed with this study, nine (9) expert and 18 novice evaluators. Thus, I organized seven (7) HE groups according to four (4) different levels of the factor presence of an expert, which ranged from no expert up to three (3) experts in the same group. Each group agreed to provide their reports for this study. Thereafter, I conducted a comparative analysis on the reports based on standard methods of the field and on a cluster analysis of similarities. I described the F-measure for each group report according to a relaxed and a strict criteria. Also, I described the dendrograms formed from the cluster analysis and the respective similarities indicated by each cluster. The results showed that the quality of reports from collaborative HE conducted by experts and novices together can be more similar to the quality of reports from a traditional HE with multiple expert inspectors (Benchmark Group) then to the quality of reports from a collaborative HE conducted by a group composed only by novice evaluators (Baseline Group). Finally, I discuss additional findings and implications for future studies in the field.A Avaliação Heurística (AH) é um método popular de inspeção de usabilidade. Entretanto, seus resultados são dependentes da experiência dos avaliadores. Este estudo explorou e descreveu a diferença na qualidade de resultados (relatórios) de AH colaborativa conduzida por grupos de avaliadores de composição distinta, considerando diferentes quantidades de avaliadores experientes em cada grupo. Vinte e sete (27) avaliadores contribuíram voluntariamente com este estudo, nove (9) experientes e 18 novatos. Assim, foram organizados sete (7) grupos de AH, de acordo com quatro (4) níveis diferentes do fator presença de avaliador experiente, variando de nenhum experiente até três (3) avaliadores experientes no mesmo grupo. Cada grupo de avaliadores concordou em entregar seus relatórios de AH para este estudo. A partir de tais relatórios, foi conduzida uma análise comparativa baseada em métodos específicos da área, e também baseado em uma análise de agrupamento com base em medidas de similaridade. Como resultado, descreveu-se as medidas F (F-measure) referentes ao relatório de cada grupo respeitando critérios estritos e relaxados de comparação. Além disto, foram descritos os dendrogramas resultados das análises de agrupamento. Os resultados mostraram que a qualidade de relatórios de AH colaborativas conduzidas por avaliadores experientes e novatos juntos pode ser mais similar à qualidade de relatórios de AH tradicional conduzida por múltiplos avaliadores experientes (Grupo Benchmark) do que à qualidade de relatórios de AH colaborativa conduzida por grupos formados apenas por avaliadores novatos (Grupo Baseline). Finalmente, discutiu-se resultados adicionais e implicações para pesquisas futuras na área.Biblioteca Digitais de Teses e Dissertações da USPFortes, Renata Pontin de MattosSalgado, André de Lima2017-08-02info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/masterThesisapplication/pdfhttp://www.teses.usp.br/teses/disponiveis/55/55134/tde-08022018-084022/reponame:Biblioteca Digital de Teses e Dissertações da USPinstname:Universidade de São Paulo (USP)instacron:USPLiberar o conteúdo para acesso público.info:eu-repo/semantics/openAccesseng2018-07-19T20:50:39Zoai:teses.usp.br:tde-08022018-084022Biblioteca Digital de Teses e Dissertaçõeshttp://www.teses.usp.br/PUBhttp://www.teses.usp.br/cgi-bin/mtd2br.plvirginia@if.usp.br|| atendimento@aguia.usp.br||virginia@if.usp.bropendoar:27212018-07-19T20:50:39Biblioteca Digital de Teses e Dissertações da USP - Universidade de São Paulo (USP)false
dc.title.none.fl_str_mv Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
Adaptações ao método de Avaliação Heurística (AH) para avaliadores novatos
title Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
spellingShingle Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
Salgado, André de Lima
Avaliação heurística
Avaliadores novatos
Efeito avaliador
Efeito experiência
Evaluator effect
Expertise effect
Heuristic evaluation
Novice evaluators
Usabilidade
Usability
title_short Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
title_full Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
title_fullStr Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
title_full_unstemmed Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
title_sort Adaptations to the Heuristic Evaluation (HE) method for novice evaluators
author Salgado, André de Lima
author_facet Salgado, André de Lima
author_role author
dc.contributor.none.fl_str_mv Fortes, Renata Pontin de Mattos
dc.contributor.author.fl_str_mv Salgado, André de Lima
dc.subject.por.fl_str_mv Avaliação heurística
Avaliadores novatos
Efeito avaliador
Efeito experiência
Evaluator effect
Expertise effect
Heuristic evaluation
Novice evaluators
Usabilidade
Usability
topic Avaliação heurística
Avaliadores novatos
Efeito avaliador
Efeito experiência
Evaluator effect
Expertise effect
Heuristic evaluation
Novice evaluators
Usabilidade
Usability
description Heuristic Evaluation (HE) is a popular method of usability inspection. However, its outcomes are dependent on the expertise of evaluators. This study explored and described the difference in quality of outcomes (reports) of a collaborative HE conducted by evaluator groups of distinct composition, regarding different numbers of expert evaluators in each group. Twenty-seven (27) evaluators voluntarily contributed with this study, nine (9) expert and 18 novice evaluators. Thus, I organized seven (7) HE groups according to four (4) different levels of the factor presence of an expert, which ranged from no expert up to three (3) experts in the same group. Each group agreed to provide their reports for this study. Thereafter, I conducted a comparative analysis on the reports based on standard methods of the field and on a cluster analysis of similarities. I described the F-measure for each group report according to a relaxed and a strict criteria. Also, I described the dendrograms formed from the cluster analysis and the respective similarities indicated by each cluster. The results showed that the quality of reports from collaborative HE conducted by experts and novices together can be more similar to the quality of reports from a traditional HE with multiple expert inspectors (Benchmark Group) then to the quality of reports from a collaborative HE conducted by a group composed only by novice evaluators (Baseline Group). Finally, I discuss additional findings and implications for future studies in the field.
publishDate 2017
dc.date.none.fl_str_mv 2017-08-02
dc.type.status.fl_str_mv info:eu-repo/semantics/publishedVersion
dc.type.driver.fl_str_mv info:eu-repo/semantics/masterThesis
format masterThesis
status_str publishedVersion
dc.identifier.uri.fl_str_mv http://www.teses.usp.br/teses/disponiveis/55/55134/tde-08022018-084022/
url http://www.teses.usp.br/teses/disponiveis/55/55134/tde-08022018-084022/
dc.language.iso.fl_str_mv eng
language eng
dc.relation.none.fl_str_mv
dc.rights.driver.fl_str_mv Liberar o conteúdo para acesso público.
info:eu-repo/semantics/openAccess
rights_invalid_str_mv Liberar o conteúdo para acesso público.
eu_rights_str_mv openAccess
dc.format.none.fl_str_mv application/pdf
dc.coverage.none.fl_str_mv
dc.publisher.none.fl_str_mv Biblioteca Digitais de Teses e Dissertações da USP
publisher.none.fl_str_mv Biblioteca Digitais de Teses e Dissertações da USP
dc.source.none.fl_str_mv
reponame:Biblioteca Digital de Teses e Dissertações da USP
instname:Universidade de São Paulo (USP)
instacron:USP
instname_str Universidade de São Paulo (USP)
instacron_str USP
institution USP
reponame_str Biblioteca Digital de Teses e Dissertações da USP
collection Biblioteca Digital de Teses e Dissertações da USP
repository.name.fl_str_mv Biblioteca Digital de Teses e Dissertações da USP - Universidade de São Paulo (USP)
repository.mail.fl_str_mv virginia@if.usp.br|| atendimento@aguia.usp.br||virginia@if.usp.br
_version_ 1809090499415375872

Registros relacionados