Sunday, June 11, 2006

WSJ on recommendation engines

Jason Fry at the Wall Street Journal looks "Under Recommendation Engines' Hood". I am briefly quoted in the article.

I particularly like this excerpt on whether recommendations should target what you think you want or what you really want:
The goal of Amazon's recommendations is simply to generate more sales than whatever would have been at the top of a customer's page without personalized information.

It's bizarre to expect Amazon to ignore the trainloads of "Star Wars" stuff I've bought from it, or to somehow detect that I thought all those Boynton books were insipid even as I kept buying them. When it comes to describing us as customers and consumers, recommendation engines may do the job better than we would.

In other words, we lie -- and never more effectively than when we're lying to ourselves ... I fancy myself a reader of contemporary literature and history books, but I mostly buy "Star Wars" novels and "Curious George" books for my kid.
Implicit data like purchases may be noisy, but it also can be more accurate. You may say you want to watch academy award winners, but you really want to watch South Park. You may say you want to read Hemingway, but you really want to read User Friendly.

The best guide to what you will want is what you wanted.

1 comment:

Anonymous said...

I think the factors (lying) that the WSJ mention are one good reason why there should be "two stages" to data gathering for personalization to overcome the whole psychology issue. What I mean is that people who fancy themselves Hemingway readers might still resist a Star Wars recommendation unless it's also presented with a Hemingway item (just because they don't want a second purchase to further admit they only buy Star Wars stuff).

In my mind the two stages are: Comparative purchase history (like the "customers who bought this also bought this") and recommendations performance ("what did customers buy from the list of recommendations").

Amazon does both ("53% of customers also bought...") which I think is a really neat way of optimizing the recommendations. If you found out that very few people bought the recommendations, you might tweak the engine to show fewer items, or use some different factors to surface more items that might be of interest.