Characterizing the value of personalized search

Jaime Teevan, Susan Dumais, and Eric Horvitz from Microsoft Research have a poster at the upcoming SIGIR 2007 conference titled "Characterizing the Value of Personalizing Search" (PDF).

Some excerpts:

Our analysis suggests that while search engines do a good job of ranking results to maximize global happiness, they do not do a very good job for specific individuals.

We observed a great deal of variation in participants' rating of results. One reason for the variability in ratings is that participants associated different intents with the same query.

Even when the [users' provided] detailed descriptions [of their intent and those descriptions] were very similar, ratings varied ... It was clearly hard for participants to accurately describe their intent.

The ratings for some queries showed more agreement than others, suggesting that some queries may be intrinsically less ambiguous.

[Current rankings tend] to be closer to the ranking that is best for the group than ... best for the individual ... A considerable gap in ... quality is created by requiring the lists to be the same for everyone.

See also my March 2006 post, "Beyond the commons: Personalized web search", that discusses another paper on a similar topic by the same authors.