Newsbreak vs. Google News

Steve Outing talks about the launch of Newsbreak, an Australian news aggregator from Fairfax Digital. The aggregator itself seems indistinguishable from many others, but what is interesting is that it comes from a traditional news organization, Fairfax Digital.

Steve says Newsbreak is a reaction to Google News and other online news aggregators. And he thinks this is a positive sign:

This continues the trend -- a good one, I think -- of traditional news organizations realizing that they can't continue to operate as islands on the Internet. Linking to other sources (even competitors, in many cases) serves the interests of readers, and establishes the news entity as a portal to the world of news, not just its own coverage. Such services give readers of a news brand less of a reason to turn to Google News, et al.

If only they would embrace the opportunity, traditional news organizations should be better positioned to innovate in online news than Google or Yahoo. Up to this point, innovation has been coming from elsewhere.

Google TV search

Stefanie Olsen at CNet reports on Google, Yahoo, and MSN's efforts on search for video streams.

One particularly interesting excerpt on Google TV search:

Google's project for TV search is ultra-secretive; only a handful of broadcast executives have seen it demonstrated so far. To build the service, the company is recording live TV shows and indexing the related closed-caption text of the programming. It uses the text to identify themes, concepts and relevant keywords for video so they can be triggers for searching.

The software allows people to type in keywords, such as "Jon Stewart," to retrieve video clips of the comedian's TV appearances, marked with a thumbnail picture with some captioning text, for example. Refining the search results for the show "Crossfire" would display a page that looks similar to a film reel, with various still images paired with excerpts of closed captioned text of the now-infamous fight between Stewart and CNN's "Crossfire" hosts. The searcher could click on and watch a specific segment of the show.

Watch out TiVo (and Comcast, DirectTV, ...).

See also my earlier post, "Query-free news search" which mentions a Google paper on searching television close caption text to find related news articles.

[via Search Engine Watch Blog]

Finding authoritative reviews

Chris DiBona digs up an interesting old article by Mimi Sheraton that criticizes Zagat's user reviews:

The Zagat surveys stand or fall on their central premise: that thousands of separate opinions add up to something like the truth ... [But] the majority can be wrong, and one well-informed opinion is worth more than those of a thousand amateurs.

It's a great point. How do you find the authoritative, well-informed, useful opinions? Not only does this apply to community-generated content like customer reviews and product ratings, but even to blog postings and discussion forum comments where the signal-to-noise ratio is equally poor.

One common approach is to allow people to rate the reviews. Amazon.com does this for customer reviews, allowing people to vote on whether the review was helpful. Slashdot takes this a step further, not only allowing users to moderate (rate comments), but also allowing users to metamoderate (rate the rating of the comment).

Mimi Sheraton would probably criticize this approach as just layering a popularity contest on top of a popularity contest. And it does have problems. For example, positive reviews on Amazon.com seem to get many more "helpful" votes than negative reviews. Slashdot moderators seem to have an adolescent sense of humor and favor ill-informed rants, perhaps seeking entertainment more than information.

So, what else can we do? Another approach is to attempt to identify authoritative people and treat all of their reviews or comments as higher quality. This is closer to what Mimi wants, well-informed reviewers to count more than uninformed reviewers. The trick is identifying informed reviewers. Amazon and Slashdot both emphasize active users, I'd guess on the theory that those that bother to put in the effort to be involved probably have something useful to say. Users could rate each other, but this again reverts into a popularity contest.

This does seem like a spot where social networks actually could be useful. Who is an authoritative reviewer? Someone who is considered authoritative by other authoritative users. Yes, it's circular, but identifying a seed set of authoritative users is enough to start the process going.

Would this work? Or would it be just another popularity contest?

Got suggestions for Findory?

Got a suggestion for Findory? We've love to hear from you.

Most e-mails to Findory are either suggestions or oh-my-god-this-is-so-great fan letters. We're thrilled by the feedback we've been getting. It's great to have such an enthusiastic and supportive community using and enjoying Findory.

If you do have ideas, suggestions, or things you'd like to see at Findory, please feel free to drop us an e-mail anytime at suggestions@findory.com or comment on this post. We'd always love to hear from you.

Mamma buys Copernic

Gary Stein (analyst at Jupiter Research) posts that Mamma, a metasearch engine based in Canada, just purchased Copernic, one of the leading desktop search companies.

Gary doesn't seem to think this was a very smart move by the so-called "mother of all search engines". He says, "Desktop Search has fully entered into the world of hype," and criticizes desktop search as having no business model: "No one's going to be cool with seeing ads -- contextual or otherwise -- displayed with their desktop results."

It also seems to me that, if Microsoft makes the default file and e-mail search on Windows "good enough" for most users -- perhaps by releasing MSN Desktop Search as part of Windows -- most of the opportunity for third-party desktop search applications will evaporate.

Update: Five months later, Copernic kills the deal due to a ongoing SEC investigation of Mamma.

The decline of web directories

Tara Calishain bemoans how Yahoo has deemphasized its web directory. Yahoo's latest redesign relegates their web directory to a corner at the bottom of the Yahoo home page.

It's an interesting point, particularly since Yahoo started as a web directory.

Google also deemphasized its directory a few months ago. Google Directory is based on DMOZ, the "largest, most comprehensive human-edited directory of the Web." At the time Google deemphasized Google Directory, I thought Google would be releasing a new, automated version of a web directory soon. That hasn't happened.

Keyword web search is great, but there's times when a browseable web directory is really useful, such as when you want a list of related sites, a comprehensive list of sites, or you're having a hard time specifying a search query that gets you what you need.

Update: Andrew Goodman says, "The lack of a definitive directory or two is the single biggest glaring hole in online search."

Newsprint is wasted on the young

Adam Penenberg at Wired says, "Newspapers Should Really Worry":

[Young] focus-group participants declared they wouldn't accept a Washington Post subscription even if it were free. The main reason (and I'm not making this up): They didn't like the idea of old newspapers piling up in their houses.

Don't think for a minute that young people don't read ... They access The Washington Post website or surf Google News, where they select from literally thousands of information sources. They receive RSS feeds on their PDAs or visit bloggers whose views mesh with their own.

In short, they customize their news-gathering experience in a way a single paper publication could never do. And their hands never get dirty from newsprint.

But should newspapers be worried? This trend toward online news is an opportunity. No longer are newspaper articles competing for scarce space on the front page and the limited space available on the newsprint. No longer are articles limited to distribution to a localized markets.

The online news audience is massive and worldwide. It's hungry for your content. All you have to do is give it to them.

[via Scripting News]

Froogle wish lists

Google launches wish lists in Froogle. Google is getting better and better for online shopping.

How do you feel about that, Amazon?

Ask Jeeves and advertising

Jefferson Graham at USA Today writes about Ask Jeeves. Some excerpts:

The company's signature cartoon butler, known as Jeeves, was a symbol of dot-com excess ... "We had great marketing, but the product just didn't deliver," [CEO Steve] Berkowitz admits about Jeeves' early days.

Jeeves was initially known for its gimmick: It promised to answer any query formed in a question. Most of the time, though, Jeeves replied with irrelevant links, sending millions away to alternatives such as Google.

[Acquiring Teoma in 2001] enabled Jeeves to acquire its own search technology and make its search results more relevant to queries ... Jeeves' most profitable move of all [was] deciding to partner with rival Google. Google-placed text ads, which appear atop Jeeves' search results, represent nearly 70% of Jeeves' income.

"We look at the Web differently — at the credibility of a source, as opposed to just the popularity of a site," says Jim Lanzone, Jeeves' senior vice president.

For instance, a search for "Bay Area airports" on Jeeves displays official airport sites for San Francisco, Oakland and San Jose. The same search on Google highlights local newspaper articles about the airports.

The biggest problem I have with Ask Jeeves is the focus on advertising. SiliconBeat illustrates this well with screenshots of the same search on Google, Yahoo, and Ask Jeeves.

On Google, search results are at the top. On Ask Jeeves, advertising (sponsored results) fill the top of the page. Which is more appealing to someone trying to find something?

Ask Jeeves' advertising-focused page may result higher short-term revenue, but Ask is crippling its long-term growth with its obnoxious and intrusive advertising.

[via Gary Price and Andy Beal]

Artist similarities in music

Brian Dennis points to an interesting paper by Brian Whitman and Steve Lawrence, "Inferring Descriptions and Similarity for Music from Community Metadata".

If that title didn't turn you off completely, the paper does have an interesting idea. Basically, they mine text in web pages, discussion groups, and blogs (which they call "community metadata") to discover information about music artists. They extract phrases from the community metadata and use it to find relationships. Because they analyze the web pages and discussion groups continuously, they claim to be able to capture short-term trends, like a groundswell of buzz around a particular song or artist.

This idea of extracting data and relationships from community metadata is clever. AllConsuming.net is an interesting example of this for books. It "watches weblogs for books that they're talking about". Memeorandum is an interesting example for news. It watches blogs to see what news articles they are talking about.

By the way, one of the authors of this paper, Steve Lawrence, is now at Google.

Findory's source pages

Findory launched another new feature this weekend, pages for every news site and blogger in our database. Alex Edelman has a good write-up.

Check out the pages for BBC, Wired, or Nature.

Take a peek at our pages for the blogs ResourceShelf, Searchblog, InsideGoogle, or Scobleizer.

Every article on Findory has a link to the appropriate source page just under the title. Each source page has recent articles, related sources, and related articles. Related sources and related articles are a great way for readers to discover interesting news stories and sources.

Try surfing from source to source using the related sources! I just used them to read recent articles from National Geographic, then clicked over to Nature, then New Scientist, then ScienceDaily.

Personalized search vs. clustering

Raul Valdes-Perez (CEO of Vivisimo) has a CNet article attacking personalized web search and touting the virtues of document clustering.

Raul makes some excellent points on the difficulties of doing personalized web search well. He says people's interests are fleeting and noisy. Raul says it's difficult to accurately infer interests from clickstream data, which is also noisy and imprecise.

It's true that personalized search is challenging. But Raul criticism is overstated. If personalized search learns immediately in response to new data, it can react to people's immediate goals and interests, even if they differ from their long-term behavior. If the personalization helps in more cases than it hurts, then the personalization has value, even if the data is noisy and the assumptions made from the data are speculative.

Raul's solution is to give up on personalized search and do document clustering instead. Vivisimo's Clusty is an excellent clustering web search -- if you haven't tried it, go try it, it's great -- but it requires effort. Users have to refine their query repeatedly using the clusters to find what they want.

People are lazy. They want what they want and they want it now. Google recognizes this, providing an "I'm feeling lucky" button that just sends you to the top search result immediately. They recognize that it's better to just find what the searcher wants on the first try, no refining, no effort.

Personalized search offers improvements to relevance rank by recognizing that relevance differs from individual to individual. Personalized search makes it more likely that you find what you need on the first try.

Robin Sloan's EPIC 2014

Robin Sloan produced a clever Flash movie called "EPIC 2004" speculating about the future of personalized news and information. Worth watching.

After a brief recap of events of the last decade, the movie speculates about a future product from Google, the Google Grid, a vast file and content-sharing network that appears to be some combination of Blogger, TiVo, Napster, and the Google cluster. In Robin's vision, this is followed by MSN Newsbotster, a personalized news site that appears to be some combination of Findory, Slashdot, Memeorandum, and social networking tools like Friendster. Next up is Googlezon and EPIC, an "evolving personalized media construct" that provide personalized information by summarizing and rewriting content dynamically for each user.

Summarizing news and documents as described in EPIC is very difficult, but it is an active area of research. One of the most interesting examples out there now is Columbia Newsblaster. Microsoft Research also is doing work in this area.

Aside from the silly brand names of Newsbotster and Googlezon, Robin Sloan has created an interesting and thought-provoking vision of the future. Definitely watch the movie.

The movie ends with criticism of this new world of personalized news and information, complaining that it will be dominated by "narrow, shallow, sensationalist trivia", apparently what Robin Sloan thinks is all people really want and all they'll get from personalized news. He also claims Googlezon and EPIC will cause the death of large and well-respected news organizations like the New York Times.

The death of the New York Times? Clearly hyperbole. At best, Google News is another distribution channel for news. While it may reduce traffic to the front page of online newspapers, it drives traffic to their content, to individual articles. As the CEO of AP said recently, the "content will be more important than its container." News organizations will continue and thrive in the future. The only differences are that content -- the work of talented reporters and writers -- will be emphasized and that the content will be distributed more widely.

Are personalized news sites more shallow or more narrow? Compare a personalized news site to the current front page of CNN. The unpersonalized front page of CNN provides only a shallow view targeting some mishmash of the general interests of millions of readers. By trying to satisfy everyone, it satisfies no one, a bland blend of interests that results in mediocrity. And, I only get the perspective of CNN, what they think is important to their readers.

Personalized news provides an opportunity to broaden reader's interests, exposing them to news sources, perspectives, and viewpoints they otherwise would never have seen. A personalized news aggregator provides both breadth and focus, sorting through huge numbers of sources and articles and helping you find what you need.

Personalized news helps you discover news you would otherwise miss. It makes it easier to get the information you need to be well-informed about the events that impact your life. If this is the future, it is a future which should excite us.

Update: A few weeks later, EPIC is making the rounds again and getting some additional coverage (CNet, Slashdot, Traffick, InsideMicrosoft). A couple folks even started a Googlezon blog.

Findory's personalized web search

A couple weeks ago, Findory launched search history for web, news, and blog search. As I've said before, search history is not personalized search.

This week, Findory took our first step toward true personalized web search. In subtle and small ways, we are starting to modify web search results based on your history at Findory.com.

To see the impact, do a web search at Findory, then click on one or two of the search results, then do another search for something fairly similar. In cases where we believe we can help, we'll modify and highlight some of the search results.

Here's a couple specific examples:

• Search for "Yahoo".
  Click on the top link for Yahoo.com.
  Search for "Dilbert".
  The Dilbert page at Google a few results down will be highlighted and modestly reranked.
  
  
• Search for "Incredibles"
  Click on the IMDB link (fourth down).
  Search for "Nemo".
  The IMDB page on Finding Nemo will be highlighted and popped up to the top slot.

Please keep in mind these are our first, early, baby steps. The changes are small, infrequent, and subtle. Findory need to learn to walk before it can run. Over time, Findory will better understand how to help people find what they need and the changes will become larger and more frequent.

As small as this step may be, we believe it is a first for a commercial web search engine. Many are talking about personalized search, but no one is doing it. Our personalized web search learns from your behavior, modifies your search results, and helps you find what you need.

Update: It took many months, but a new version of Findory personalized web search has launched that makes more substantial changes in the relevance rank.

Google Kirkland open house

The Google Kirkland open house party was last night. It was a great time. Quite a turnout, totally packed. Strong UW presence, which wasn't surprising, but I was amazed by the number of Amazon and MSN people there.

Brady Forrest (PM at MSN Search, frequent poster on MSN Search blog) was there. Scott Pitasky (former Amazon.com, now head of HR at MSN). Erik Selberg (author of Metacrawler, one of the first metasearch engines, now at MSN Search). Robert Scoble was apparently there, but I didn't bump into him.

I got a chance to catch up with a few Googlers, Joshua Redstone (old friend from graduate school, works on GFS), Peter Norvig, Jeff Dean. Jeff Dean and I had an interesting discussion about the potential for abuse of MapReduce; I was arguing you might see tragedy of the commons issues because the system makes it so easy to consume vast resources on the Google cluster, but Jeff said everyone plays nice and that it isn't an issue. I was hoping to see Joe Beda, but he couldn't make it, unfortunately. David Krane was there, but I didn't see him.

Bumped into a couple of the Slashcode guys too, Brian Aker and Chris Nandor. Unbelievable that Slashdot uses NFS in a production system, but Brian and Chris insist it's not a serious problem.

I finally got a chance to meet Todd Bishop from the Seattle PI in person. Great to see you there, Todd.

Making sense of the chaos

Bill Joy (co-founder, Sun Microsystems) on the Charlie Rose show:

Our lives are overwhelmed by all the information coming at us in a very disorganized way. We're going to hunger for something that will make sense of all the chaos--that will look at all the things happening in the world and filter and order them in a way that's personalized to us. That will be the next great revolution--that is something that doesn't take an index of the dead information on the Net, but the live information of things as they are occurring and as they are relevant to us.

The next great revolution is finding focus and relevance in the flood of new information. The next great revolution is personalized news.

[via Musing on Technology]

AI is the mainstream

Adam Bosworth (who left BEA for Google recently) at his ICSOC 2004 talk:

You want to see the future. Don’t look at Longhorn. Look at Slashdot. 500,000 nerds coming together everyday just to manage information overload. Look at BlogLines. What will be the big enabler? Will it be Attention.XML as Steve Gillmor and Dave Sifry hope? Or something else less formal and more organic? It doesn’t matter. The currency of reputation and judgment is the answer to the tragedy of the commons and it will find a way. This is where the action will be. Learning Avalon or Swing isn’t going to matter. Machine learning and inference and data mining will. For the first time since computers came along, AI is the mainstream.

Managing information overload with AI. That's the future.

[via Niall Kennedy]

Google, Glog, and G-nius

The Motley Fool writes about Google's track record of innovation.

In addition to coining some odd words like "g-nius" and "glog", the article has an interesting piece on Google AdWords and AdSense:

In 2003, Google derived 97% of its revenues from advertising.

Google's simple text-based ad results often segue so well with the search that they can hold actual interest for the user. Even the most grudging critic probably has to acknowledge that there was a time when he or she clicked on one of Google's ads because of its relevance to their interests.

Relevance. That's where Google's got it down, especially with its AdWords and AdSense programs.

See also my earlier post, "Bringing sense to web advertising".

[via Andy Beal]

The big, bad VC

An interesting essay, "10 Reasons to Shy Away from Venture Capital".

The essay echoes advice I've gotten from many others. The distraction and poorly aligned goals are the biggest issues for me. We're too busy executing to play these games.

[via Joel on Software]

Intent marketing

Charlene Li at Forrester writes about personalized advertising, which she labels "intent marketing":

Today, publishers announce that they have content and an audience that is attracted to that content – so if you're a marketer interested in that audience, the publisher will sell you access to those users in the form of advertising at a specified price. The onus falls to the marketer to figure out where the audience is, hence the important role of media buyers and ad agencies.

In the future, marketers will announce that they want to reach a certain segment – let's say, women in-market for a car – and are willing to pay $25 per qualified lead. The onus now falls to the publisher to deliver that audience to the marketer. Publishers will be able to see what the "bids" are within the system for a particular user profile and optimize their ad serving to maximize revenue per page.

This is the development of what I call "intent marketing" where the marketer targets intent, in this case, inferred from past behaviors.

I'd like to see this go one step further. I'd like to see the entire process of targeting advertisements handled as an optimization problem.

In this future, marketers create a large pool of advertisements with specific segments in mind for each ad. The advertisements go out on the network of publishers, mostly showing to people who match the segments, but also sampling related segments outside of the marketers intent. Quickly, the advertisements focus in on narrow clusters of readers who are interested or, if no one seems interested, the advertisements are dropped completely.

I'm not alone in having this vision. Many have talked about it. But it's quite a challenge to implement. It requires a massive amount of data, only possible at scale.

But Google AdWords seems close to doing it. They suggest alternative keywords, show ads for queries that aren't exact matches to the specified keywords, and drop ads that perform poorly. The next step is to use the vast amount of data they have on what ads are effective to start showing ads for other keywords than what was specified and to further narrow the targets when responsive audiences are found.

Already, I click on Google ads much more than other ads because they're relevant, especially when I do a Google search for a specific product. Perhaps advertising actually can be informative, unobnoxious, and useful.

See also my earlier post, "Bringing sense to web advertising".