Towards Simulated Feedback


Abstract:

Relevance feedback has received wide attention recently, as a means to capture the search context either explicitly or implicitly to improve search accuracy. While explicit feedback is more reliable than implicit feedback, it is difficult to obtain because of the extra effort needed which makes implicit feedback an interesting and easy to obtain alternative. Most search engines already have implicit user feedback consisting of logs of user interactions over the web which is a very valuable source for research related to user feedback. However, such feedback is usually not available to public or even research communities at large for reasons like being a potential threat to privacy of web users. This makes it difficult to experiment and evaluate web search related research and especially web search personalization algorithms. Also, due to dynamic content of the web and rapidly changing ranking algorithms of major web search engines, explicit and implicit feedback collected earlier may become stale. Given these problems, we are motivated towards an artificial way of creating user feedback, based on insights from query log analysis. We call this simulated feedback. We believe that simulated feedback can be immensely beneficial to web search engine and personalization research communities. by greatly reducing efforts involved in collecting user feedbacks. The benefit from “Simulated feedback” are - It is easy to obtain and also the process of obtaining the feedback data is repeatable and customizable. In this paper, we describe a simple yet effective approach for creating simulated feedback. Creation of simulated feedback is done in two steps. In the first step, a simulated user is created and in the second step we simulate web search process and create simulated feedback. We have evaluated simulated feedback by comparing it with implicit feedback available from query logs and with explicit feedback from judges and achieved 60% accuracy and 66% accuracy respectively.