Saturday, February 17, 2007

Relevance at Google, Yahoo, and Microsoft

Gord Hotchkiss at Search Engine Land has an interesting post where he summarizes his conclusions after interviewing members of the usability groups at Google, Yahoo and Microsoft.

Some selected excerpts:
At Google, relevance is a religion. The Golden Triangle is sacred. Nothing can appear here if it's not, in Google's judgment, absolutely relevant to the user.

Both Yahoo and Microsoft said similar things, but in each case, the importance of relevance was always counterbalanced with other factors. At Google, relevance is the only thing that mattered.

The urge to monetize is winning the battle at Yahoo. Yahoo was by far the most aggressive both in terms of how often top sponsored ads were shown, and how much [space] was devoted to them ... Relevance, at least in terms of the user's expectation, was pushed down the page.

The religion that is relevance at Google is not found to the same extent at Microsoft ... Relevance does not define the core purpose of Microsoft ... Microsoft approaches the user experience as one of a number of best practices ... It's not that relevance isn't important at Microsoft. It's just that it's treated more as a business objective than a sacred cow.

I believe its Google's obsession with relevance that makes the difference. Relevance drives a better user experience, which drives market share, which drives monetization. Google seems to have the best understanding of this basic cause-and-effect relationship in the search marketing ecosystem.
I think the history of MSN and Yahoo as portals may be part of the explanation here.

Google has never had a portal. Google has always focused on getting people the information they need quickly and then sending them off to other sites to do whatever they need to get done.

Microsoft, AOL, and Yahoo have large websites with a lot of content. They traditionally have wanted to capture people on their properties and keep them there. Because of this history, I suspect Microsoft and Yahoo may still be conflicted about whether their goal in search is helping people get what they need quickly or capturing audiences for their sites.

Gord's thoughts on where Google, Yahoo, and Microsoft may be going with search are also interesting:
Google is heading the push towards greater personalization of search results. It's an essential next step to reach the ultimate objective, which is to give their user exactly what they're looking for at any given time.

Expect Yahoo to start exploring ways to thread ... individual user experiences together with search as the common element. They'll look for ways in which Yahoo's community can help enhance the search experience and provide social context to it. How this will be accomplished as a little cloudy at this point.

Microsoft's approach to the future of search seems to be the most fragmented of all the three major players .... [but] I would expect Microsoft to start using the vast amount of data that's available to them in the form of click stream behavior across all their properties, and begin to use this not only as an advertising targeting opportunity, but also to help improve the relevance of search results. From this, further personalization is the logical next step.
Google is focused on using better algorithms, big data, and personalization to improve relevance. Yahoo is focused on social search, which is poorly defined but seems to mean building tools to help humans help humans find what they need. Microsoft is scattered, but probably will stumble toward personalization at some point. All sounds about right.

For more on these topics, see also some of my earlier posts, "Google dominates, MSN Search sinks", "Google expands personalization" and "Social software is too much work".

Update: There is a fun debate on search personalization going on in the comments for this post. Don't miss it.


Anonymous said...

My interpretation of Yahoo's social search is that they are using humans as the bridge between different pieces of information. Generally, they're following the collaborative filtering path. And if so, that is fairly well defined. That's my take, at least.

I have still yet to really understand what Google means by, and wants to do with, personalization. To me, personalization in an or Findory context makes a lot of sense.. because that is recommendation personalization. But personalization in a search context really seems strange. Search is an entirely different beast from recommendation.

Actually, in our discussion a few days ago about Powerset, NLP hype, and Danny Sullivan, I reread Danny's original October 2006 NLP critique.

And I thought to myself: Y'know, you could apply that exact same critique to search personalization! First, here is what Danny has to say:

What conceptual analysis are you going to do [on the query] "pirates of the carribbean." We can guess that a user might have these topics in mind:

* the first movie
* the second movie
* a computer game
* the ride at disneyland
* actual pirates that roamed the carribbean

There are many more meanings and intentions. But natural language analyze the four words "pirates of the caribbean" all you want. You won't know exactly what concept someone has in mind from that analysis.

Now, replace references to NLP with references to personalization: What personalization are you going to do [on the query] "pirates of the carribbean"?

We can guess that a user might have one of the above four topics in mind. But even if you know, based on all the personalized information that you have accrued about the user, that the user has seen the first movie and not the second, and you know that the person is almost never interested in historical characters or entities, and that the person has never been to Disneyland, how do you know which answer to give?

Since the person has seen the first, but not the second movie, does that mean they are looking up the first movie again, to buy it on DVD? Or does it mean that they are trying to find showtimes for the second movie, because they haven't seen it yet? Or is this one of the rare instances where they actually are interested in carribean piracy? Or did they just win the Superbowl (something you won't find in their search history) and are going to Disneyland?

It seems to me that many if not most of the same critiques you can leverage against NLP you can also leverage against personalization. How will personalization help us for the "pirates of the carribean" query?

Or for the "beach" query? If you see from the the person's search history that they like to visit the ocean, does that mean they are looking for information on the beaches that they have already visited? Or ones that they have not yet visited? It could really go either way. And sometimes the user might mean one, and sometimes they might mean the other. So how does personalization and user search history really help you know what the user is after, this very moment?

Note that Danny concludes that the way out of the NLP conundrum is to go with a tools such as Clusty/Vivisimo, in which the various aspects of the query are presented to you, and with a single click you can indicate which aspect you mean. I would argue, indeed I have been arguing, that the same Clusty approach is the solution to the personalization conundrum, too. You really don't know if they mean the first movie or the second movie, no matter what is in their search history. Therefore you cannot rely on that search history, and instead should take the Clusty interactive approach, instead.

IMHO :-)

Greg Linden said...

How will personalization help us for the "pirates of the carribean" query?

Hi, Jeremy. Good question there.

I don't mean to sound flippant, but I think it helps when it helps.

Specifically, I think using your history and the history of other users helps when it indicates that searchers similar to you are more likely to be interested in a particular search result than the norm.

For example, lets say that, just before you searched for [pirates of the carribean], you searched for [disneyland] and we have evidence that past searchers with that pattern are much more likely to be interested in the Pirates ride at Disney than the norm. Then, we could try emphasizing that result and seeing it if it helps people find what they need faster. If it does, we keep emphasizing it.

Or, to take another example, lets say a profile built on months of search history show a strong interest in computer games and that others with a strong interest in computer games that do a search for [pirates of the carribean] tend to be most likely to click on the game link. We could bump that link up higher and see if it helps. If it does, we keep the change for that group of users, otherwise we toss it.

It is true that, in both of these cases, you really do not know the user intent. But you have an estimate of the probability of different intents and how much that likelihood differs from the norm. That is enough that personalization could be able to help many people much of the time.

There is a substantial difference between Danny's critique of NLP and your attempt to apply the same critique to personalization. Danny is complaining that NLP does not have enough information to disambiguate a query. Personalization adds more information to a query -- your search history and comparisons with everyone else's search histories -- and that additional information is what gives it promise.

Anonymous said...

No, you don't sound flippant at all. "It works when it works" sounds quite reasonable, actually.

What I am struggling to understand is when it really would work. When it works. And how you avoid making sure that when it doesn't work, you at least do no worse than generic, non-personalized results. Because I think that is a real danger. One can overlook a bad recommendation, because that information wasn't one's primary goal, anyway. But a misinterpreted search personalization result is the kiss of death.

You do a fine job of explaining the rough mechanics of how one might actually implement a personalization solution. And you've really got me into that geek-groove brainstorm mode, where it would be fun to toss a bunch of ideas around.

But with the examples you give, I don't see why you would need personalization. Take the Disneyland example. If I do a search for Disneyland, and then a "personalized" search for Pirates, I should(!) get the same results as if I had done a "search within these results" for Pirates, off of my initial Disneyland query. So why bother with history logging or fancy personalization algorithms or anything like that? I just modify my original search.

"Well," you respond, "what if Disneyland wasn't your immediately previous query.. what if you searched for Disneyland a lot, in the past.. just not two minutes ago. Then you could personalize your current Pirates query toward other people who also searched for Disneyland," right?

I guess this is where I am still not seeing the big picture. Because it seems to me that not only will I have a lot of Disney in my search history, but being the typical American media consumer that I am, I will have a lot of video games and movies in my history, too. And maybe even some carribean geography, too, from that vacation I was planning a few years ago. How does the personalization algorithm know which parts of my search history to tie the current "Pirates" query to?

This is why, I think, we can draw similarities to Danny's NLP critique. Superficially NLP and personalization may be different; NLP has too little data and personalization has much more. But with the latter, you still don't know which aspect of the user's search history the current query is expanding on or drawing from! D'ya see what I mean? Or am I really not getting it?

Anonymous said...

I have a more detailed followup to my last comment. However, rather than asking you to respond, Greg, I would be curious to hear other comments from some of your other readers. I hope this is not inappropriate for me to call out your lurkers, because this is your blog, not mine. But to me, Web 2.0 as much about the conversation as it is about the blog, and I am interested in knowing what everyone else thinks, too.

As I was saying, personalization still has the same bottleneck as NLP; it just comes in at a different point in the algorithm. You still have to decide which part of one's history to use. And which parts of everyone else's history to tie it to. And at that point, there is just as much uncertainty about this many-to-many mapping as there is with the NLP query. Yes, what you say about using probabilities is good. But to use probabilities, you have to first estimate probabilities. So what I am talking about are the binning functions, or the kernel density estimators, or whatever other non-parametric count aggregation mechanisms you are using, to create those probabilities in the first place. How do you know which parts of which query histories to tie together?

An example with more detail: I like video games. Both PC and console. I often search for them, trying to find reviews and screenshots. I don't tend to search much for movies, though I do see them from time to time. My search history would probably peg me as a "gamer".

But as much as I enjoy video games, I severely dislike games based on movies. These "tie-in" games tend to be pretty shoddily made and unexciting, banking more on the popularity of the movie than on good gameplay. So even though my search history is full of games, there is no chance I would ever query for the Pirates game.

Given my search history for video games, how is the personalization going to know that I do mean the video game and not the creature when I type "lemmings" but that I mean the movie and not the video game when I type "pirates"? I appear to be in the "gamer group", right? And movie games sell enough that there are probably going to be lots of clicks from the "gamer" group to the Pirates game. Personalization would therefore move me toward the game. And be dead wrong. That's dangerous, because personalization would actually degrade the results.

So will the personalization algorithm have to be modified, so that it somehow learns that Jeremy is in the gamer group, except when the game is a movie game, in which case Jeremy is in the movie group?

What about when the movie comes out after the video game? Aren't they making a Halo movie? Like many, I enjoyed the fps xbox game. But the Halo movie just sounds awful, and there is no way I'd see it. So let's suppose I now do a query for Halo. By this time the personalization engine will have learned that I mean video game, except when there is a movie, because of the whole Pirates of the Carribean movie/game thing. As a result, personalization will give me the Halo movie, and not the game!

But with Halo, since the movie is based on the game, and not the other way around, I actually do mean the game and not the movie, when I search for Halo.
So will the personalization engine have to learn an exception to the exception to the rule, and nudge it back toward the game again, if and only if the game came out before the movie?

If this feels like it is getting complicated, it is. The problem with search personalization is that you have to make decisions about group membership where, in reality, there are too many exceptions and too much fluidity around one's information needs. I think dimensionality reduction would probably even exacerbate the problem, rather than solve it.

Like you, I remain enthusiastic but skeptical. It would be good to see search personalization (which I think is different from recommendation personalization like Findory and Amazon) succeed, but at the moment I do not believe the hype.

Anonymous said...

My view on personalized search is that the weight of the personalization in the ranking of results should be low (especially on the first page of results) except in the case of very recent search history (help the user refine an unsuccessful search).