Monday, February 27, 2006

Different visions of the future of search

Ingrid Marson at ZDNet nicely captures the differences between the visions of MSN, Yahoo, and Google for the future of web search.

Saleel Sathe (Lead PM, MSN Search) argues that searchers will have to change their behavior and learn how to use search better:
"Search engines have shot themselves in the foot by providing a search box, where users provide relatively little information," Sathe said.

"The average search query is 2.3 words... but if you asked a librarian for information you would not just give them 2.3 words -- you would give them the opportunity to give you the rich detailed answer you want."
Bradley Horowitz (head of Technology Development Group at Yahoo Search) argues that we should throw out existing web search and replace it with social search:
"Where is the next big breakthrough that gets beyond PageRank? PageRank confers a privilege to Webmasters who vote by proxy for all of us."

"What we think is the next major breakthrough is social search. It basically democratises the notion of relevance and lets ordinary users decide what's important for themselves and other users," said Horowitz.
Matthew Glotzbach (Director, Google) argues that the computer should do the work and figure out what people need from whatever information is available:
"In the distant future we will not be able to get you to take more action ... We will get close enough with what you give us. A lot of emphasis will continue on doing that in the background -- getting the technology to figure out [what you want]," [Matthew] said.

"Larry Page [the co-founder] of Google often says, 'the perfect search engine would understand exactly what you mean and give back exactly what you want'."
MSN (and, until recently, A9) wants to give you more powerful tools. Yahoo wants the community of users to help improve search. Google wants computers to do all the work to get you what you need.

But all of these strategies face formidable challenges.

There certainly is promise in treating search as a dialogue -- an iterative process rather than a one-shot deal -- but I think any attempt by MSN to get users to do more work is doomed from the start. People are lazy, appropriately so. They want what they want and they want it now. If you don't find it for them quickly and easily, they'll switch to a tool that will.

Yahoo's social search, as I've said before, faces two major hurdles: spam and non-participation.

Spam, oh glorious spam. Letting ordinary users decide what others see is great until those ordinary users discover the profit in promoting their own sites. Whether the wisdom of the crowd can overpower the hucksters of the bazaar remains to be seen.

And these lofty notions of replacing a supposed "vote by proxy" tyranny of webmasters with a democracy of widespread participation will succumb to the reality that, again, users are lazy, and few will participate if it requires effort. At best, vote by one proxy will be replaced by vote by another proxy, something Bradley himself has acknowledged.

But, Google's chosen path is a daunting one as well. It is very hard to figure out what you want in the face of limited, noisy information about what you actually want. The key to this likely will be to use all information available, including your history of searches, but development of scalable, high quality, personalized search is still in the baby steps of its infancy.

No matter how you look at it, it's an exciting time for search. Much will happen in the coming years.


Michael Fagan said...

why "until recently" for A9?

Greg Linden said...

Well, Udi Manber, who argued for forcing people to learn how to search better, left A9 recently.

I'm not sure whether they'll continue down their old path or look for a new strategy now.

More on Udi and A9 in a few of my previous posts ([1] [2] [3]).

Michael Fagan said...

Well I left A9 before Udi did (internship ended, not because I don't like A9), but I wouldn't necessarily assume that one person (okay, so he was teh CEO ;-) ) leaving will change everything.

Greg Linden said...

My understanding is that the only reason A9 exists is because of Udi, so I think he was pretty key.

Anonymous said...

Brilliant post, Greg. I've always had this gut feeling that these were GYM's three different philosophical approaches to web IR. But to see them side by side like this is enlightening and clarifying, and really drives home the point. Your commentary on each is good, too.

I will, however, take issue with one of your thoughts: There certainly is promise in treating search as a dialogue -- an iterative process rather than a one-shot deal -- but I think any attempt by MSN to get users to do more work is doomed from the start. People are lazy, appropriately so. They want what they want and they want it now. If you don't find it for them quickly and easily, they'll switch to a tool that will.

I think what people tend to forget is that, when the search doesn't work the first time, we are already doing much more work. We are already not lazy.

Think about it. I'll bet you a hunnert to one that, when using Google and not finding what you were looking for, you have refined your search, and issued the query again. Maybe you added a word. Maybe you took a word away or changed it to a synonym. Maybe you put a phrase in quotation marks. Or added a "+" operator in front of some particular word. But the point it, you reissued your query, did you not?

Well, all that is meant by Microsoft in "search as a dialog" is that, instead of you going back and refining your query over and over and over again, you let the search engine offer clarifying suggestions to you, from which you can choose. Click the one you meant, and you automatically refine your query, and get at what you want.

So how can users not want such a tool, but instead be happy to refine and reissue the query themselves? Why is it that, after refining the query 3-4 times and finally getting what they want, they say "wow, Google is amazing! There is my result, at the very top of the list!" Don't users realize that they're already doing all this work themselves?

The problem is I don't think they realize it. And I don't think they blame it on the search engine. I recall reading a study once (can't remember details, so take this comment with a grain of salt) that suggested that when Google didn't work, users blamed themselves. So they consciously didn't think of the 3-4 iterations it took until they issued the "right" query as part of their search.

Well, I'm sorry. If Google is relying on this.. on users blaming themselves, and reissuing queries until they "get it right", then their overall experience is just going to get worse and worse, while companies that actually let me easily refine my query (in dialog form) are going to get better and better.

Greg Linden said...

Good point, Jeremy. I agree that suggestions on query refinements are an important part of search as a dialogue. It is a good way to help people find what they want.

Offhand, I'd say Clusty is probably the best example of a search engine pursuing that strategy.

Anonymous said...

Greg I think Google may also move toward Social Search, even if it does not abandon its current approach.
See my post IS Google Contemplating Social Search at
What do you think?

Anonymous said...

Since searchers are no better at putting together a search query now than they were six or seven years ago, it would seem prudent to just put that dream out of our minds. And if we expect the computer to do all the work of figuring out that when we type the query “horses” in a search box when we’re really looking for “saddles”, then I’d say we’re setting our goals a little high. I can’t get Google to figure out that where I’m from “bass cleaning” is not cleaning a musical instrument, so my hopes are low that it’ll do much better anytime soon.

Naw, I’d say that Bradley Horowitzis on to something there. Maybe the two giant worlds, Search and Social Networks, are about to collide. I recently saw this on CNN Money, "...As such, Bill Tancer, general manager of global research with Hitwise, an Internet research firm, thinks Google needs to make more deals in social networking. He said Google's biggest threat will not come from search engines like Yahoo!, Microsoft's MSN or IAC/Interactive's but from MySpace, the social networking site owned by News Corp.

"Google's core business is helping people consume information on the Web. As we see Google grow out its content offerings such as finance and maps, I think they'd have to make a serious play into more social and user-generated content," he said..."."

MySpace claims around 67 million profiles, Friendster around 24 million, Tagworld, Tagged, Orkut, LiveJournal, etc., etc., all add up to well over 100 million users, maybe significantly more. Extending the power of these types of huge networks into pure search could be a major factor in the evolution of Search. The huge communities, and the smaller sub-communities within these sites could influence virtually all the key factors in search, including relevance, inclusion and exclusion, advertising paradigms, and the actual content itself. Could it, infusing the right social search technologies, eventually take the place of (or be strongly complementary to) traditional search technologies? It’s starting to look like the answer is “yes”. Sure, we'll have to improve the way we search tags and the process of tagging itself. Also, we have to make saving and sharing link information an organic part of a user's browsing experience, and more... But, at such a scale, human-indexed search results might be superior to traditional algorithmic search results, and ordered in a superior way (by the community). And with millions of members probably sharing around a million links per day, the index would be large enough to compete with anyone within a few months. It could be a significant. Why would the social networks be interested in playing? This article in the NYT illustrates that other avenues of monetizing their site will be on their radar moving forward.