Tuesday, May 01, 2007

Personalized search yields dramatically better search results

Greg Sterling at Search Engine Land, in his article "iGoogle, Personalized Search and You", quotes Google VP Marissa Mayer as saying:
[Personalization is] one of the biggest relevance advances in the past few years.

Personalization doesn't affect all results, but when it does it makes results dramatically better.
See also excerpts from a few recent articles by Search Engine Land columnist Gord Hotchkiss in my earlier post, "Personalization, Google, and discovery".


jeremy said...

I went and read the Search Engine Land article, and beyond these claims I found no discussion of evidence that backed these statements up. While I am extremely pleased to hear that personalization only helps and never hurts, I would really like to hear about some of the evaluation that went into this conclusion.

How does it help, exactly? Has the average position of the first relevant document gone from 20 to 10? That is a 100% improvement in precision, and would be great, because now the user no longer has to go to the second page to find a relevant document. Or has the average position of the first relevant document gone from 2 to 1? That is also a 100% improvement in precision, but probably not as useful or significant, because I usually see the correct result right away, anyway.

Or has personalization helped, because users are thrashing less, i.e. issuing fewer re-queries, after their original query has failed? If it has cut down the number of queries users have to do, to find the information they seek, that is also significant.

But I see no details from Google as to why it's actually better. Any idea whether they have talked about those at all? Or are they simply telling us "it's better"?

Greg Linden said...

Hi, Jeremy. I agree, it would be great to hear the metrics behind Marissa's claim that it is "dramatically better".

I cannot do much to elucidate the data behind Google's claim, but I can point to a very limited test Findory was able to do on its personalized web search that showed that "about 3% of the time, the reordering moved the clicked result to the top slot or one of the top 5 slots when it was not there already."

Tiny little Findory was already able to get better search results from personalized search when its personalization affected the results. I assume that Google, with all their resources and data, was able to achieve much more.