Monday, November 20, 2006

Creating a smart Google

Jeffrey O'Brien at Fortune writes about "The race to create a 'smart' Google". Some excerpts:
Recommender systems ... are sprouting on the Web like mushrooms after a hard rain. Dozens of companies have unveiled recommenders recently to introduce consumers to Web sites, TV shows, other people - whatever they can think of.

The company that can decipher all that information ... will pinpoint your tastes and determine the likelihood that you'll buy any given product. In effect, it will have constructed the algorithm that is you.

There's a sense among the players in the recommendation business ... that now is the time to perfect such an algorithm.

The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.
I couldn't have said it better myself. You cannot search for something if you don't know it exists. Discovery helps surface interesting gems without any effort, without any explicit search, from a sea of information.

There is a good quote in the article from John Riedl, someone who has been working on recommender systems longer than just about anyone:
"The effect of recommender systems will be one of the most important changes in the next decade," says ... professor John Riedl ... "The social web is going to be driven by these systems."
A good friend, Brent Smith, is also quoted:
Amazon realized early on how powerful a recommender system could be and to this day remains the prime example. The company ... [compares] your purchasing patterns with everyone else's and thus narrow a vast inventory to just the stuff it predicts you'll buy.

"Personalized recommendations," says Brent Smith, Amazon's director of personalization, "are at the heart of why online shopping offers so much promise."
The article does focus on promise, taking a negative tone toward well established, lucrative systems at companies like Amazon and Netflix but giving startups, some of which have little more than vaporware, the benefit of the doubt.

It is a little unfortunate. The article leads with a sensationalistic title -- that Google sure ain't that smart, heh, heh, snark, snark -- but then fails to show anything that clearly represents progress toward a smarter Google. In fact, after name dropping Udi Manber and Peter Norvig, the article even holds up Google as the likely leader in the race to build a smarter Google.

But, overall, I agree that recommender systems are growing in importance, especially in terms of application to web search, advertisements, and video, and that future recommendation systems will be even more lucrative than they are now. As a good colleague of mine was fond of saying, "The future will be personalized."

Update: Mike at TechDirt doesn't like the hype either, and then goes a step further by slamming all recommender systems as "far from useful", "exceptionally limited", and "littered with failures". While I think it is going too far to condemn all recommender systems -- I am not sure Mike is aware of how much money personalization features generate for, for example -- his post is good for a contrarian view.


fishzle said...

Search is a highly visible part of the internet, but it's not what people spend all their time doing. Look at myspace, the various photo sites, the millions of personal home pages people have. Search may morph into "discovery", but if search is only a 4-5% activity, then what's the big deal? People spend a whole lot more time reading and writing email via hotmail than search or probably anything else. Email is the key, enhancing the mail reading experience is where the buzz should be.

Anonymous said...

"how much money personalization features generate for"

humm a clue for that ?? Thanks for sharing it !

Anonymous said...

I found your blog back in June after I got a strange book recommendation from Amazon:

Dear Customer,

We've noticed that customers who have purchased Perl Best Practices by Damian Conway also purchased books by David Yack. For this reason, you might like to know that David Yack's ASP.NET 2.0 MVP Hacks is now available in paperback.

I started thinking that maybe these recommendation programs might be able to determine trends before they were evident to the public. Were people abandoning Perl and Linux for Microsoft products? Was the job market influencing a shift to ASP.NET? Or, was Microsoft reacting to those Mac ads on tv and trying to be cooler?

But then I tried going to the Amazon page for Damian Conway's book and following the links under 'Customers who bought this item also bought,' and strangely never hit on anything by David Yack.

Don't know the answers yet, but I'll keep reading.