Word embedding-based representations for short text

• Autor(a) principal: Marcelo Rodrigo de Souza Pita
• Data de Publicação: 2019
• Tipo de documento: Tese
• Idioma: eng
• Título da fonte: Repositório Institucional da UFMG
• Texto Completo: http://hdl.handle.net/1843/38885; https://orcid.org/0000-0001-7582-4651
• topic: Short text topic modeling; Short text representation; Word vectors; Computação – Teses; Modelagem de tópicos – Teses; Representação de textos - Teses; Processamento de linguagem natural (Computação) – Teses; Aprendizado de máquina – Teses

Short texts are everywhere in the Web, including social media, Q&A websites, advertisement text, and an increasing number of other applications. They are characterized by little context words and a large collection vocabulary. This makes the discovery of knowledge in short text challenging, motivating the development of novel effective methods. An important part of this research is focused on topic modeling that, beyond the popular LDA method, have produced specific algorithms for short text. Text mining techniques are dependent on the way text is represented. The need of fixed-length input for most machine learning algorithms asks for vector representations, such as the classics TF and TF-IDF. These representations are sparse and eventually induce the curse of dimensionality. In the level of words, word vector models, such as Skip-Gram and GloVe, produce embeddings that are sensitive to semantics and consistent with vector algebra. A natural evolution of this research is the derivation of document vectors. This work has contributions in two lines of research, namely, short text representation for document classification and short text topic modeling (STTM). In first line, we report a work that investigates proper ways of combining word vectors to produce document vectors. Strategies vary from simple approaches, such as sum and average of word vectors, to a sophisticated one based on the PSO meta-heuristic. Results on document classification are competitive with TF-IDF and show significant improvement over other methods. Regarding the second line of research, a framework that creates larger pseudo-documents for STTM is proposed, from which we derive two implementations: (1) CoFE, based on the co-occurrence of words; and (2) DREx, which relies on word vectors. We also propose Vec2Graph, a graph-based representation for corpora induced by word vectors, and VGTM, a probabilistic short text topic model that works on the top of Vec2Graph. Comparative experiments with state of the art baselines show significant improvements both in NPMI and F1-score.