Limits to surprise of recommender systems

• Autor(a) principal: André Paulino de Lima
• Data de Publicação: 2019
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Biblioteca Digital de Teses e Dissertações da USP
• Texto Completo: https://doi.org/10.11606/D.100.2019.tde-15042019-175412

Surprise is an important component of serendipity. In this research, we address the problem of measuring the capacity of a recommender system at embedding surprise in its recommendations. We show that changes in surprise of an item owing to the growth in user experience, as well as to the increase in the number of items in the repository, are not taken into account by the current metrics and evaluation methods. As a result, in so far as the time elapsed between two measurements grows, they become increasingly incommensurable. This poses as an additional challenge in the assessment of the degree to which a recommender is exposed to unfavourable conditions, such as over-specialisation or filter bubble. We argue that a) surprise is a finite resource in any recommender system, b) there are limits to the amount of surprise that can be embedded in a recommendation, and c) these limits allow us to create a scale up in which two measurements that were taken at different moments can be directly compared. By adopting these ideas as premises, we applied the deductive method to define the concepts of maximum and minimum potential surprises and designed a surprise metric called \"normalised surprise\" that employs these limits. Our main contribution is an evaluation method that estimates the normalised surprise of a system. Four experiments were conducted to test the proposed metrics. The aim of the first and the second experiments was to validate the quality of the estimates of minimum and maximum potential surprise values obtained by means of a greedy algorithm. The first experiment employed a synthetic dataset to explore the limits to surprise to a user, and the second one employed the Movielens-1M to explore the limits to surprise that can be embedded in a recommendation list. The third experiment also employed the Movielens-1M dataset and was designed to investigate the effect that changes in item representation and item comparison exert on surprise. Finally, the fourth experiment compares the proposed and the current state-of-the-art evaluation method in terms of their results and execution times. The results obtained from the experiments a) confirm that the quality of the estimates of potential surprise are adequate for the purpose of evaluating normalised surprise; b) show that the item representation and comparison model that is adopted has a strong effect on surprise; and c) indicate an association between high degrees of surprise and negatively skewed pairwise distance distributions, and also indicate a significant difference in the average normalised surprise of recommendations produced by a factorisation algorithm when the surprise employs the cosine or the Euclidean distance