Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects
Autor(a) principal: | |
---|---|
Data de Publicação: | 2015 |
Outros Autores: | , , |
Tipo de documento: | Artigo |
Idioma: | eng |
Título da fonte: | Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
Texto Completo: | http://hdl.handle.net/1822/33092 |
Resumo: | Background and Objectives Document annotation is a key task in the development of Text Mining methods and applications. High quality annotated corpora are invaluable, but their preparation requires a considerable amount of resources and time. Although the existing annotation tools offer good user interaction interfaces to domain experts, project management and quality control abilities are still limited. Therefore, the current work introduces Marky, a new Web-based document annotation tool equipped to manage multi-user and iterative projects, and to evaluate annotation quality throughout the project life cycle. Methods At the core, Marky is a Web application based on the open source CakePHP framework. User interface relies on HTML5 and CSS3 technologies. Rangy library assists in browser-independent implementation of common DOM range and selection tasks, and Ajax and JQuery technologies are used to enhance user-system interaction. Results Marky grants solid management of inter- and intra-annotator work. Most notably, its annotation tracking system supports systematic and on-demand agreement analysis and annotation amendment. Each annotator may work over documents as usual, but all the annotations made are saved by the tracking system and may be further compared. So, the project administrator is able to evaluate annotation consistency among annotators and across rounds of annotation, while annotators are able to reject or amend subsets of annotations made in previous rounds. As a side effect, the tracking system minimises resource and time consumption. Conclusions Marky is a novel environment for managing multi-user and iterative document annotation projects. Compared to other tools, Marky offers a similar visually intuitive annotation experience while providing unique means to minimise annotation effort and enforce annotation quality, and therefore corpus consistency. Marky is freely available for non-commercial use at http://sing.ei.uvigo.es/marky |
id |
RCAP_27e92d1855480412be7e0644b1caf826 |
---|---|
oai_identifier_str |
oai:repositorium.sdum.uminho.pt:1822/33092 |
network_acronym_str |
RCAP |
network_name_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository_id_str |
7160 |
spelling |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projectsDocument annotationCollaborative annotationIterative annotationInter-annotator agreementTracking systemScience & TechnologyBackground and Objectives Document annotation is a key task in the development of Text Mining methods and applications. High quality annotated corpora are invaluable, but their preparation requires a considerable amount of resources and time. Although the existing annotation tools offer good user interaction interfaces to domain experts, project management and quality control abilities are still limited. Therefore, the current work introduces Marky, a new Web-based document annotation tool equipped to manage multi-user and iterative projects, and to evaluate annotation quality throughout the project life cycle. Methods At the core, Marky is a Web application based on the open source CakePHP framework. User interface relies on HTML5 and CSS3 technologies. Rangy library assists in browser-independent implementation of common DOM range and selection tasks, and Ajax and JQuery technologies are used to enhance user-system interaction. Results Marky grants solid management of inter- and intra-annotator work. Most notably, its annotation tracking system supports systematic and on-demand agreement analysis and annotation amendment. Each annotator may work over documents as usual, but all the annotations made are saved by the tracking system and may be further compared. So, the project administrator is able to evaluate annotation consistency among annotators and across rounds of annotation, while annotators are able to reject or amend subsets of annotations made in previous rounds. As a side effect, the tracking system minimises resource and time consumption. Conclusions Marky is a novel environment for managing multi-user and iterative document annotation projects. Compared to other tools, Marky offers a similar visually intuitive annotation experience while providing unique means to minimise annotation effort and enforce annotation quality, and therefore corpus consistency. Marky is freely available for non-commercial use at http://sing.ei.uvigo.es/markyThe authors thank the project PTDC/SAU-ESA/646091/2006/FCOMP-01-0124-FEDER-007480FCT, the Strategic Project PEst-OE/EQB/LA0023/2013, the Project "Bio-Health - Biotechnology and Bioengineering approaches to improve health quality", Ref. NORTE-07-0124-FEDER-000027, co-funded by the Programa Operacional Regional do Norte (ON.2 - O Novo Norte), QREN, FEDER, the project "RECI/BBB-EBI/0179/2012 - Consolidating Research Expertise and Resources on Cellular and Molecular Biotechnology at CEB/IBB", Ref. FCOMP-01-0124-FEDER-027462, FEDER, and the Agrupamento INBIOMED from DXPCTSUG-FEDER unha maneira de facer Europa (2012/273). The research leading to these results has received funding from the European Union's Seventh Framework Programme FP7/REGPOT-2012-2013.1 under grant agreement no. 316265 (BIOCAPS) and the [14VI05] Contract-Programme from the University of Vigo. This document reflects only the author's views and the European Union is not liable for any use that may be made of the information contained herein.ElsevierUniversidade do MinhoPérez-Pérez, MartínGlez-Peña, DanielFdez-Riverola, FlorentinoLourenço, Anália2015-022015-02-01T00:00:00Zinfo:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/articleapplication/pdfhttp://hdl.handle.net/1822/33092engPérez-Pérez, Martín; Glez-Peña, Daniel; Fdez-Riverola, Florentino; Lourenço, Anália, Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects. Computer Methods and Programs in Biomedicine, 118(2), 242-251, 20150169-260710.1016/j.cmpb.2014.11.00525480679info:eu-repo/semantics/openAccessreponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãoinstacron:RCAAP2023-07-21T12:49:59Zoai:repositorium.sdum.uminho.pt:1822/33092Portal AgregadorONGhttps://www.rcaap.pt/oai/openaireopendoar:71602024-03-19T19:48:35.921223Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãofalse |
dc.title.none.fl_str_mv |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
title |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
spellingShingle |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects Pérez-Pérez, Martín Document annotation Collaborative annotation Iterative annotation Inter-annotator agreement Tracking system Science & Technology |
title_short |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
title_full |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
title_fullStr |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
title_full_unstemmed |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
title_sort |
Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects |
author |
Pérez-Pérez, Martín |
author_facet |
Pérez-Pérez, Martín Glez-Peña, Daniel Fdez-Riverola, Florentino Lourenço, Anália |
author_role |
author |
author2 |
Glez-Peña, Daniel Fdez-Riverola, Florentino Lourenço, Anália |
author2_role |
author author author |
dc.contributor.none.fl_str_mv |
Universidade do Minho |
dc.contributor.author.fl_str_mv |
Pérez-Pérez, Martín Glez-Peña, Daniel Fdez-Riverola, Florentino Lourenço, Anália |
dc.subject.por.fl_str_mv |
Document annotation Collaborative annotation Iterative annotation Inter-annotator agreement Tracking system Science & Technology |
topic |
Document annotation Collaborative annotation Iterative annotation Inter-annotator agreement Tracking system Science & Technology |
description |
Background and Objectives Document annotation is a key task in the development of Text Mining methods and applications. High quality annotated corpora are invaluable, but their preparation requires a considerable amount of resources and time. Although the existing annotation tools offer good user interaction interfaces to domain experts, project management and quality control abilities are still limited. Therefore, the current work introduces Marky, a new Web-based document annotation tool equipped to manage multi-user and iterative projects, and to evaluate annotation quality throughout the project life cycle. Methods At the core, Marky is a Web application based on the open source CakePHP framework. User interface relies on HTML5 and CSS3 technologies. Rangy library assists in browser-independent implementation of common DOM range and selection tasks, and Ajax and JQuery technologies are used to enhance user-system interaction. Results Marky grants solid management of inter- and intra-annotator work. Most notably, its annotation tracking system supports systematic and on-demand agreement analysis and annotation amendment. Each annotator may work over documents as usual, but all the annotations made are saved by the tracking system and may be further compared. So, the project administrator is able to evaluate annotation consistency among annotators and across rounds of annotation, while annotators are able to reject or amend subsets of annotations made in previous rounds. As a side effect, the tracking system minimises resource and time consumption. Conclusions Marky is a novel environment for managing multi-user and iterative document annotation projects. Compared to other tools, Marky offers a similar visually intuitive annotation experience while providing unique means to minimise annotation effort and enforce annotation quality, and therefore corpus consistency. Marky is freely available for non-commercial use at http://sing.ei.uvigo.es/marky |
publishDate |
2015 |
dc.date.none.fl_str_mv |
2015-02 2015-02-01T00:00:00Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/article |
format |
article |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://hdl.handle.net/1822/33092 |
url |
http://hdl.handle.net/1822/33092 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.relation.none.fl_str_mv |
Pérez-Pérez, Martín; Glez-Peña, Daniel; Fdez-Riverola, Florentino; Lourenço, Anália, Marky: a tool supporting annotation consistency in multi-user and iterative document annotation projects. Computer Methods and Programs in Biomedicine, 118(2), 242-251, 2015 0169-2607 10.1016/j.cmpb.2014.11.005 25480679 |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.format.none.fl_str_mv |
application/pdf |
dc.publisher.none.fl_str_mv |
Elsevier |
publisher.none.fl_str_mv |
Elsevier |
dc.source.none.fl_str_mv |
reponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação instacron:RCAAP |
instname_str |
Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
instacron_str |
RCAAP |
institution |
RCAAP |
reponame_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
collection |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository.name.fl_str_mv |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
repository.mail.fl_str_mv |
|
_version_ |
1799133064117354496 |