Evaluation of performance of a software of automatic sumarization of texts

• Autor(a) principal: Tabosa, Hamilton Rodrigues
• Data de Publicação: 2020
• Outros Autores: Souza, Osvaldo de, Cândido, José Carlos dos Santos, Melo, Ana Cristina Azevedo Ursulino, Reis, Keila Giullianna Braga
• Tipo de documento: Artigo
• Idioma: por
• Título da fonte: Informação & Informação
• Texto Completo: https://ojs.uel.br/revistas/uel/index.php/informacao/article/view/35928
• topic: Automatic Summarization of Texts; Access to Information; Natural Language Processing; Mediation (Practice); Sumarización Automática de Textos; Acceso a la Información; Procesamiento del Lenguaje Natural; Mediación (Práctica); Sumarização Automática de Textos; Acesso à Informação; Processamento da Linguagem Natural; Mediação (Prática)

Intrudoction: Since 2014 we have developed a research to produce a software (prototype) that would be able to elaborate automatic summaries of texts based on techniques of Natural Language Processing and frequency statistics of words. The first empirical tests of the tool generated results that indicated a significant reduction of the dimensionality of the texts, with considerable preservation of their semantic value. Objective: In this article, we present the results of the continuity of our investigative work, based on a human evaluation of the quality of these abstracts from blind tests. Metodology: A group of three librarians received a mixed and unidentified block of abstracts - produced by humans and the automatic abstracts made by the software - and carried out an evaluation, according to the criteria of grammatical correctness, preservation of central ideas, coherence and readability, extension of abstract, whether there was paraphrase or copy of fragments, and if there was introduction of ideas not contained in the original text. Results: The results showed that in four of the five evaluation criteria adopted, there was a qualitative equivalence between the abstracts produced by humans and those produced by the software, which seems to represent a relative success since the prototype could replace a person in the resume activity texts without leaving anything to be desired, except in the fifth evaluation center, referring to the dimension of the abstract, in which the text produced by the software was pointed out as extensive beyond what was necessary. Conclusions: Despite the good results of the prototype, we realized the need for improvements in its performance, as well as to evaluate it by more comprehensive methods, from more representative samples and by a larger group of evaluators.