This research work extends the use of document features in some earlier work on RankNet with new "user action" features.
However, every searcher sees the same search results in the system. Eugene said that "personalization is a very interesting direction for future research," but that this system "does not do personalization."
I apologize. I was too quick to link this paper with some of Susan Dumais' other work.
Microsoft researchers Eugene Agichtein, Eric Brill, and Susan Dumais have a paper
The paper reports some good successes with reordering search results based on
We show that incorporating user behavior data can significantly improve ordering of top results in real web search setting.
We show that incorporating implicit feedback can augment other features, improving the accuracy of a competitive web search ranking algorithms by as much as 31% relative to the original performance.
The general approach is to re-rank the results obtained by a web search engine according to observed clickthrough and other user interactions for the query in previous search sessions. Each result is assigned a score according to expected relevance/user satisfaction based on previous interactions.
In table 6.1 in the paper, there is one result I find particular interesting. They got a lot of lift merely from increasing the rank of any document on which the searcher clicked in the past (referred to as "BM25F-RerankCT"). That is, without learning a user model at all, it helps quite a bit to simply favor what the searcher clicked on before.
Most of the paper focuses its time on methods for learning which document features are most of interest to a given searcher. It is a complicated approach that, I suspect, will struggle against the problems of sparse data and over-generalization.
Given the success of just increasing the rank of anything I clicked on before,
By the way, if you like this MSR paper, you might also be interested in another paper by the same authors to appear at the same conference, "Learning User Interaction Models for Predicting Web Search Result Preferences" (PDF).