Creating a smart Google

Jeffrey O'Brien at Fortune writes about "The race to create a 'smart' Google". Some excerpts:

Recommender systems ... are sprouting on the Web like mushrooms after a hard rain. Dozens of companies have unveiled recommenders recently to introduce consumers to Web sites, TV shows, other people - whatever they can think of.

The company that can decipher all that information ... will pinpoint your tastes and determine the likelihood that you'll buy any given product. In effect, it will have constructed the algorithm that is you.

There's a sense among the players in the recommendation business ... that now is the time to perfect such an algorithm.

The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.

I couldn't have said it better myself. You cannot search for something if you don't know it exists. Discovery helps surface interesting gems without any effort, without any explicit search, from a sea of information.

There is a good quote in the article from John Riedl, someone who has been working on recommender systems longer than just about anyone:

"The effect of recommender systems will be one of the most important changes in the next decade," says ... professor John Riedl ... "The social web is going to be driven by these systems."

A good friend, Brent Smith, is also quoted:

Amazon realized early on how powerful a recommender system could be and to this day remains the prime example. The company ... [compares] your purchasing patterns with everyone else's and thus narrow a vast inventory to just the stuff it predicts you'll buy.

"Personalized recommendations," says Brent Smith, Amazon's director of personalization, "are at the heart of why online shopping offers so much promise."

The article does focus on promise, taking a negative tone toward well established, lucrative systems at companies like Amazon and Netflix but giving startups, some of which have little more than vaporware, the benefit of the doubt.

It is a little unfortunate. The article leads with a sensationalistic title -- that Google sure ain't that smart, heh, heh, snark, snark -- but then fails to show anything that clearly represents progress toward a smarter Google. In fact, after name dropping Udi Manber and Peter Norvig, the article even holds up Google as the likely leader in the race to build a smarter Google.

But, overall, I agree that recommender systems are growing in importance, especially in terms of application to web search, advertisements, and video, and that future recommendation systems will be even more lucrative than they are now. As a good colleague of mine was fond of saying, "The future will be personalized."

Update: Mike at TechDirt doesn't like the hype either, and then goes a step further by slamming all recommender systems as "far from useful", "exceptionally limited", and "littered with failures". While I think it is going too far to condemn all recommender systems -- I am not sure Mike is aware of how much money personalization features generate for Amazon.com, for example -- his post is good for a contrarian view.