Thursday, December 08, 2005

Yahoo Answers and wisdom of the crowd

Jeremy Zawodny has the post on the Yahoo Search blog announcing a new product, Yahoo Answers.

Jeremy describes it as "a place to tap the collective wisdom of the Internet for advice, recommendations, theories, jokes, ... whatever."

Both Gary Price and Michael Bazeley talk about the similarity of the new Yahoo Answers with existing forums and message boards. I think this comparison is pretty accurate.

There are already moderated discussion forums where people can rate the quality of posts. Yahoo Answers would appear to be essentially the same thing, a user-moderated forum for people to talk about whatever.

Both Gary and Michael also contrast Yahoo's offering with Google Answers. Yahoo Answers is a free service where anyone can answer a question. Google Answers is a paid service where expert researchers answer questions. This difference is important.

Google Answers keeps quality high by charging a fee and restricting who can answer a question. Yahoo Answers hopes to keep quality high with a rating and reputation system.

Unless Yahoo Answers' reputation system includes something novel that does a better job of ferreting out experts, the site will have the same problem all user-moderated forums have. A popularity contest isn't the best way of getting to the truth.

People don't know what they don't know. Majority vote doesn't work if people don't have the information they need to have an informed opinion.

There was a case in the news a couple years ago of a legal advice site that had user-moderated forums. The idea was that lawyers would come on to the site, give short opinions, and use the goodwill gained to drum up future business.

A teenage kid with no legal training whatsoever hopped on to the system and started answering hundreds of legal questions with common sense answers. Despite the fact that some of his advice was wrong, badly wrong in some cases, he had the highest ratings on the site.

There is wisdom in the crowd. There is also a lot of noise. Separating the wisdom from the noise is the challenge.

Update: Looking at the Yahoo Answers point system, it appears to me that there is an incentive to answer as many questions as possible as quickly as possible without worrying about accuracy. I think that's going to need some tuning.

Update: Gary Price points out that Ask Jeeves had a very similar system to Yahoo Answers called AnswerPoint that they shut down in 2002. Why did they shut it down? Ask Jeeves SVP Jim Lanzone told Gary that the user base was very small, that "as a free service, there was little incentive for people to answer other people's questions," and that "it was usually just faster and easier for people to search normally ... than to submit a question to the community and wait for an answer."

Update: Nine months later, Philipp Lenssen posts an interview with a frequent contributor to Yahoo Answers named Michael. Michael said that, on Yahoo Answers, "the signal-to-noise ratio is astounding ... it's very difficult to sort the wheat from the chaff." He also disliked the Yahoo Answers point system, saying that they "encourage people to just give one-liner spammed responses to questions instead of actually putting in some thought."

5 comments:

Anonymous said...

wasn't a legal advice site, it was askme.com.

http://www.umass.edu/cyber/readings/marcusarnold.htm

has a copy of the NYT Mag article.

Joe McCarthy said...

At the MIT Enterprise Forum panel on search last night, one of the highlights (for me) was a question from a man who is president of a company that produces inhalers, and the problems he has encountered as a result of "experts" he has hired to help optimize his company's searchability (but who have violated certain searchability guidelines by the major search engines). It was a highlight both because it was an emotional and passionate statement in an event that was otherwise largely devoid of such peaks, and because it helped highlight a very important problem that was otherwise not effectively addressed (how do legitimate content producers "get found" -- I'm reminded of Robert Fulghum's exhortation to a hide-and-seek player who didn't quite understand the goal of the game: "Get found, kid!" ).

I mention this in the current context because I have doubts about any company, even Google, acting as an arbiter of expertise across a broad array of areas, and am more optimistic about the "wisdom of crowds" approach adopted by Yahoo -- with its reputation system(s) -- producing a more sustainable ecosystem.

For example, I find myself paying less attention to the editorials and publisher reviews of books on Amazon, and paying more attention to user reviews (and the ratings of the reviews and reviewers) when making purchasing decisions. As Eckart Walther (VP of Yahoo Search) observed, the web is one huge, live experiment, so we'll have the opportunity to see soon enough which model works better.

Greg Linden said...

Thanks, Anonymous, for the link to the article! Yep, that's the one.

Steve Gordon said...

I can comment a bit on both subjects. I developed Allexperts.com, the first large scale free Q&A site. It actually worked quite well (and may still do so--owned by the NY Times now), but required close oversight and a good user-inputted rating system. I found that with enough ratings, the system worked.

For the person writing about Amazon user reviews, he's right to suggest that objective information about books is better than marketing hype. I now run a service called Allreaders.com (www.Allreaders.com) which provides over 1.5 million users every month with objective information about the plots, setting, characters and themes of books--with no information on subjective questions, such as whether a book is "great" or not.
--Steve Gordon.

T.A. McCann said...

Very good discussion. As the founder of www.helpshare.com we debate points vs. $ and its relationship to quality vs. quantity of questions all the time. Clearly there is a need for the right mix of Convienence, Credibility (rating) and Compensation (reward). Depending on the goals of the site/community, a different currency method should be employed. We have implemented our sites using $, which gets higher quality answers and really incents the experts to do the work, but we are sure that $ limits traffic. In a discussion with TIm O'reilly earlier this week, we debated this fact. We are planning to do a points based version of HelpShare for some clients (we brand our service for existing communities) and it will be intersting to see the contrast