Named entity error prediction across domains

• Autor(a) principal: Andrade, Juliana Valpasso de
• Data de Publicação: 2023
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)
• Texto Completo: http://hdl.handle.net/10451/63160
• topic: Domínio/Área Científica::Humanidades::Línguas e Literaturas

This thesis was conducted within the scope of Machine Translation (MT) as part of an internship at the Portuguese company Unbabel, a startup that uses MT and other Artificial Intelligence (AI) services. In this work we propose to analyse the Named Entity Error Prediction (NEEP) model developed by the company, which also involves a community of translators contributing to error annotation and post-editing of MT. Over the past decade, much research has focused on Named Entities (NE), whether in the domain of Named Entity Recognition (NER) systems for their identification (Nouvel, 2016) or within the Multidimensional Quality Metrics (MQM) framework (Lommel, 2014), with the goal of classifying them accurately to enhance MT processes for NEs. Therefore, the main objective of this thesis is to analyse the first version of the NEEP model (v1) and, based on this analysis, explore paths for improving the model through data augmentation to retrain it. This process led to the development of a new model version, known as NEEP v2, which underwent testing and demonstrated substantial performance improvements. Understanding the process of NE annotation and error annotation in NEs was a fundamental part of the research methodology, which is divided into two parts. The first part entails describing the two datasets used first for training the initial version of the NEEP model and, subsequently, for testing it. The second part details the data augmentation process, focusing on constructing a new dataset for retraining the NEEP model to address the most common NE errors. From the results we obtained throughout the first analysis, we gained insights into the main areas requiring improvement in the NEEP (v1) model. Building a dataset for data augmentation primarily involved inducing errors in the NEs that were most error-prone. In this regard, we used data from different clients in various domains and languages to generate these new errors, using smaug - a multilingual data augmentation package that focuses on altering specific aspects of sentences, such as NEs - and manual induction of NE errors. After the retraining phase, we obtained a new version, called NEEP v2, and subsequently selected a sample of 3,164 segments to test the new model. Considering it was our initial attempt to create a model capable of identifying NE errors in MT outputs, we believe we achieved successful outcomes. We successfully addressed significant NE error issues across various languages and domains, as evidenced by our initial assessment of the NEEP v1 model and the subsequent analysis of the NEEP v2 model following the data augmentation we proposed. In conclusion, we believe that the study of both versions of the NEEP model has significantly contributed to Unbabel. All the research process led to the development of a new model version (NEEP v2), that is able to work in quite different language pairs and domains. Its prominent F-measure results (over 20 F-measure increased performance) also represent a remarkable gain and sets light for further investigations towards achieving increasingly satisfactory model results.