Using dwell times for search relevance

UNC Professor Diane Kelly gave a Google engEdu talk recently titled "Relevance Feedback: Getting the Most out of Your User".

I found the discussion of implicit relevance feedback in search -- using click and display times to determine whether a page was relevant -- that starts around 33:00 to be particularly interesting.

One noteworthy tidbit is that Diane says (at 58:00) that their work found "no direct relationship between display time and usefulness" of web pages.

I was surprised to hear that. I thought dwell times were a good indicator of the relevance of a page.

There appear to be more details on that in her 2004 SIGIR paper, "Display time as implicit feedback: Understanding task effects" (PDF). That paper, in a small user study, found "great variation between subjects in the relationship of display time and usefulness rating ... [and a] lack of any significant relationship between these two factors ... [which] indicates that using display times averaged over a group of users as a measure of usefulness is unlikely to work."

Please see also Joachims et al. 2005, "Accurately interpreting clickthrough data as implicit feedback", which discusses (in Section 2) this study and other studies on the usefulness of the time spent reading a page for determining relevance.

Please see also my earlier posts, "Actively learning to rank" and "Ranking using Indiana University's user traffic".