A relevance rank for news and weblogs

If you use a feed reader like Bloglines, there must have been at least a few times you've looked at the overwhelming pile of unread articles with a sigh. So much to read.

All feed readers organize the articles in the same way. They group the articles by feed and sort the articles by date. So, you go through, click on each feed, skim the articles, and slog on through.

"Wouldn't it be nice," you've probably thought, "if these articles were sorted by relevance? Maybe the most important articles at the top and least important at the bottom? Then I could just read the articles from top to bottom, stopping when I get bored or run out of time."

That would be nice. But, what does it mean? What's the most relevant news?

Let's explore it. What if all the articles from the news and weblog feeds were sorted by how many people read them? The more people have read an article, the higher in your list of unread articles.

Hmm... That might help, but it'd be ordered by popularity, not relevance. Yahoo News has an example of ordering news by popularity. You can see that it tends toward the sensationalistic and tabloid. It pulls you toward the mainstream and away from the long tail. That's the wrong direction, folks. You want interesting and useful, not bland and mediocre.

Okay, if it's not most popular, what is the most relevant news?

Maybe the problem is that we're defining popularity too broadly. Does it matter to me if a teenage surfer chick thought an article with rumors of Britney Spears' pregnancy was really awesome? Not in the slightest. Does it matter if one of my computer geek friends really enjoyed an article on the upcoming MySQL 5 release? Yes, that does matter.

So, perhaps relevance is what people like me like. Okay, so I'll just list hundreds of people I know who are like me, get them all to use the same feed reader, and then... oh, shucks, that's never going to happen, is it?

Fortunately, it doesn't have to. We can find people like me, people I don't even know, automatically and anonymously using some clever algorithms. Put that computer to work, I say.

Great! Now we know how to sort news by relevance. We take all the news and sort by what people like me like. So, why isn't anyone doing this? Well, someone is doing it -- and doing it quite well, I might add -- but why isn't anyone else?

Well, it's hard. Really hard. Maybe I made it sound easy, but the devil is in the details. For example, the most interesting articles for a subgroup isn't actually the same as the most popular; it's a little different, and that's just one of tens of spots where you can trip up and hork the quality of the relevance rank. These "clever algorithms" I mentioned can be really expensive; doing this at scale for millions of readers requires a lot of careful thought. News is perishable -- old news is no news -- so you better find a good solution to the cold start problem. And the list goes on. It's not easy.

But it's got to be done. It takes too long and too much effort to use the current generation of feed readers. To break into the mainstream, next generation feed readers will have to sort articles by relevance.

Search engines and blog search

Gary Price posts about IceRocket's new weblog search. It's a good example of how web search products easily can be extended to offer blog search as well.

How long before the search giants offer weblog search? Ask Jeeves already supports it through Bloglines (and, very soon I'd expect, through their main page).

What about Google, Yahoo, and MSN? What's stopping them from launching blog search?

Profile of del.icio.us

Alarm:clock posts an interesting profile of del.icio.us, the popular social bookmarking site.

See also the recent Slashdot article on de.lirio.us, an open source competitor to del.icio.us.

Piling on Google

Om Malik rips Google and praises Yahoo.

Ben Hammersley at the Guardian claims Google "has been overtaken" and that "Yahoo is the new Google."

Danny Sullivan responds, rebutting many of the points and providing some much needed perspective.

Desktop search should not exist

Rob Pegoraro at the Washington Post reviewed desktop search tools. He ends his article by saying:

These programs also shouldn't exist: Their capabilities should be built into the operating system, something both Microsoft and Apple are working on.

Exactly right. The opportunity for third-party desktop search apps exists because the Microsoft Windows file search is pitifully weak. As soon as Microsoft corrects this flaw, the opportunity will evaporate, as will the numerous also-ran desktop search apps.

[via Marc Orchant]

The best of the old and the new

Dan Gillmor posts that newspapers are "trapped -- highly profitable businesses that can't or won't take the kind of risks that will be crucial to survival." He's looking for "ways to combine the best of the old and the new" to help newspapers and ensure a vibrant future for journalism.

Dan's right that newspapers are trapped. Newspapers used to have a local monopoly on distribution of news, local advertising, and classifieds that generated extraordinary profits. Online sites have created a disruptive new distribution channel. Some newspapers are responding by grasping for their fading monopoly -- throwing up barriers to online and restricting access to their content -- but that will only cause it to slip away faster.

Instead, newspapers should focus on their advantage, a unique understanding of their local market. No one can do local reporting better than newspapers. No one has more experience dealing with local advertisers than newspapers. By focus on their competitive advantage, newspapers can benefit from new distribution channels.

For example, the San Jose Mercury News should be the world experts on Silicon Valley and Silicon Valley businesses. The Seattle PI should be the world experts on the Puget Sound, including Boeing and Microsoft. Newspapers should focus on what they do better than anyone else. They should embrace the new distribution channels to provide their content to the world.

I'm not alone in believing this is the future. Tom Curley (CEO, AP) made a similar prediction at the Online News Association Conference a few months ago.

So, how can the online media help newspapers? Online news aggregators like Findory provide one example. Findory shows short excerpts of news content from thousands of sources. We direct readers to the newspaper websites to read the full articles.

We see ourselves as having two customers, the readers of our sites and the providers of our news. We help our readers find interesting and relevant news. We help newspapers find readers for their content. We are new media helping old media and old media helping new. The combination makes it easier for readers to get the information they need.

Another day, another deal

Michael Bazeley reports that Google acquired Urchin, a web analytics company.

The deals are coming fast and furious lately. Gannett, Knight-Ridder, and Tribune acquired Topix.net. IAC acquired Ask Jeeves. Ask Jeeves acquired Bloglines. Yahoo acquired Oddpost, Farechase, and Flickr. MSN acquired Lookout. Google acquired Keyhole and Picasa. Looksmart acquired Furl.net. Mamma acquired Copernic. Six Apart acquired LiveJournal. Whew!

Update: SiliconBeat reports on two more Google acquisitions, Zipdash and Where2.

Personalized TV advertising

Lorne Manly at the New York Times writes about the future of television ads, personalized advertising:

The television commercial -- a blunt instrument that often reaches as many disinterested people as desired ones -- is beginning to behave like a smarter version of direct mail. Ads can be customized, not just by neighborhood, but ultimately by household and perhaps by viewing habits.

If you don't own a dog, you won't be bombarded by ads for Puppy Chow or Iams. If the technology determines that the man of the house has wrested control of the remote from his teenage daughters, he will not have to sit through feminine hygiene ads during the most popular network shows.

Their ultimate goal is ... to make ads more relevant to the lives of viewers, so that they might just stick around and watch. Instead of commercials being an annoyance, they become information a viewer needs, perhaps even craves.

I've seen two major trends lately as businesses try to improve the effectiveness of advertising. Some are making ads more intrusive and more obnoxious, hoping to make them more difficult to ignore. Others are making ads more relevant, hoping that people will find them useful.

If you view advertising as being propaganda -- trying to hoodwink you into buying something you don't want or need -- then this new trend toward targeting may be disturbing. After all, targeted, effective propaganda is worse than untargeted, ineffective propaganda.

But if you view advertising as information -- useful information about products and services you don't know about and might want -- then this new trend toward targeting will seem positive to you. It will waste less of your time and help you get the information you need.

Behind the scenes at Findory

Sometimes the impact of our work at Findory is obvious. When we launch a new feature, such as source pages, everyone immediately can see what we've done.

But sometimes our work is more subtle. Most of our work this month was on the backend systems. Sure, you might notice that the site is a little faster, but the effect is subtle, especially since we already served our personalized pages in under 100 ms. If you were very observant, you might notice that we partitioned off our RSS traffic to new servers, but since it all just works like it's supposed to, you probably wouldn't notice. And that's okay. We like it when it all just works.

But work on the backend systems can be satisfying. We substantially improved the quality of the unpersonalized Findory front page recently. Sure, Findory is about personalized news, but we want newbies to Findory to immediately see interesting and useful content, and our latest improvements will make Findory more attractive to new users. It's satisfying when we make a change that we think helps our readers find the news they need.

We completely rewrote our deployment system. When you have a cluster of servers, you want to be able to image the servers and put new content on them quickly and easily. This allows you to scale easily and rapidly as new traffic floods in. Our old system was a little too manual and unreliable for our taste, so we made it nice and bulletproof.

Ahh, yes, deployment systems. Both Alex and I have done this before. Way back in 1997, Amazon.com ran on one big piece of iron, a multiprocessor DEC Alpha box. It might be hard to believe now when clusters are seen as the obvious solution, but, in 1997, web server clusters weren't all that common. I worked on a team of two to split Amazon's website across a cluster of four boxes, a job that mostly involved ferreting out assumptions that there was only one box, but that also included writing the deployment system to push and pull content from the boxes. It was the first step toward Amazon's current cluster architecture, and it was exciting to be on the bleeding edge of it. More recently, Alex wrote a deployment system for some of Amazon's famous web services. We know scalable systems, and it's fun to be building them again at Findory.

Finally, we've been fielding a lot of interest in Findory from the search giants, newspapers, and VC firms. Perhaps some of this is that the exponential growth we've been seeing is now visible in less accurate sources like Alexa. Perhaps some of this is the growing realization that the future of news is personalization. Either way, we're excited to be part of that future.

Early peek at Yahoo 360

A few reviews are starting to come in from a closed beta of Yahoo's upcoming social networking and blogging site, Yahoo 360. Charlene Li says:

Central to the whole service is the concept that you want to communicate and connect with the people that you already know, rather than try to meet new people. To this end, your home page on the service shows the most recent content published by people within your network ... In essence, the content is being pushed to you by the service.

Curiously, it sounds a lot like Amazon's About You pages.

But I think Yahoo is on to the right idea. Social networking is needs a purpose. It has to be about more than just building a network. I'm not sure whether sharing Yahoo content is a compelling enough purpose, but at least it has a purpose.

Some reviews of Yahoo 360 seem to say that it doesn't pass the "Would your mother use this?" test. Danah Boyd said, "The controls are really overwhelming" and "I'm really worried about the novice user."

Complexity is a problem with a lot of social networking sites, and it's too bad that Yahoo didn't solve this problem. Ideally, using the site should be fun, effortless, and feel like play, not work.

Update: Looks like invites are flying for Yahoo 360. I spent some time on it. It's like some combination of Orkut, MSN Spaces, and Amazon's About You pages.

I'm enjoying playing with it, but I am finding setting it up to be a lot of work, perhaps too much of a startup hurdle for many users. It's got nice integration with some blogging and messaging tools. We'll see if that's enough of a purpose to keep people coming back.

Seeing Yahoo 360 makes me think about missed opportunity for both Amazon and Google. Amazon has had About You pages for 4-5 years but didn't expand them further into a general social software tool. Google has had Orkut and Blogger for a couple years, but we've seen little additional innovation from them.

Update: Two and a half years later, Yahoo 360 shuts down.

Personalization is hard. So what?

Philipp Lenssen lists some problems with doing personalized search. Boiling it all down, Philipp is saying that personalized search is hard to do well and can't ever be perfect, so it's not worth doing.

He's right that personalization is hard. It has to work from noisy, sparse information. It has to deal with changing preferences. It has to avoid pigeonholing. It has to make good predictions. And it has to do it all in real time for millions of users.

And he's right that it can't ever be perfect. Personalization is so hard that it's going to make mistakes. Probably a lot of mistakes.

Does this mean personalization is useless? Of course not. Personalization doesn't have to be perfect. It just has to be good enough.

If personalization helps people find what they need faster on average, it's a win. It doesn't have to be right all the time. It just needs to be helpful.

Take one of the examples from Philipp's critique, someone who searches for the single word "restaurant". If those results aren't targeted to a best guess at your location, they're completely useless. Try it on Google and look at the top results. In the vast majority of cases, it would be more helpful to emphasize your local restaurants than return the generic results. Personalization would be helpful.

Or let's take Amazon.com's personalization. Amazon's personalization is far from perfect, but it doesn't have to be. A generic storefront emphasizing top sellers is much less useful to you than a storefront emphasizing mostly products you like. When Amazon's personalization guesses wrong, it shows you something you didn't want, which is what the generic storefront would have done anyway. In general, their personalization is helpful.

Personalization is hard. Personalized search will make mistakes. And that's okay. It doesn't have to be perfect. It just has to be helpful.

Personalized search at PC Forum

Dan Farber at ZDNet reports on the search panel at PC Forum. Some excerpts on personalized search:

Google's [Marissa] Mayer is looking at providing more personalization capabilities. "We don't know how to do [personalization] well, so we are starting with baby steps, such as knowing where you are as a context."

"We need to get better not at doing searches, but at providing answers people are looking for, " [said Marissa.] "There will be a day when ten HTML links regardless of who you are is not the answer any more." She also said that the idea of everybody getting the same search result isn't reasonable.

Mayer [said] that Google's goal isn't to force users to have to think about search ... One of the Google principles is that it your "mother" can't figure out how to use a feature, it shouldn't be released.

Google has it exactly right. Personalization is hard. Figuring out how to do it well isn't obvious or easy. But it has to be done to get further improvements in relevance rank. It has to be done to help people find what they need.

Udi Manber (CEO, A9) appears to have a different view. Udi is quoted as saying, "People will learn to use search better but have to invest the thinking -- we are not in the mind reading business."

I disagree. Searchers will not do extra work unless they see immediate, extraordinary, and obvious results from doing more work.

People are lazy, appropriately so. If you need to read minds to prevent them from having to do work, well then you better read minds. They'll think it's your fault, not theirs, if you don't give them what they need.

Dan ends his article with this prediction:

The last five years of search history was more about monetization and continuity than delivering more relevance and personalization ... The next five years might yield more in the way of personalized answers.

[ZDNet article via Gary Price]

Update: Turns out the transcript for this talk is available as a PDF file after free registration on the site. Interesting reading. [via Paul Kedrosky]

Update: Four years later, Udi Manber (now at Google) appears to have changed his mind, saying, for example, that if users can't spell or don't know how to search, it is Google's problem.

Writing code at Google

Joe Beda posts some interesting details about the development environment at Google. Some excerpts:

It is really easy to look at and contribute to code in other projects without having to talk to anyone.

Teams are actively encouraged to share the most intimate details of their projects with the rest of the company.

When someone comes up with a new idea, the most common response is excitement and a brainstorming session. Politics and who owns what area rarely enter into it.

This is the way it should be. Everyone should constantly be seeking to improve everything. There should be no sacred cows. Assumptions should be questioned. Ideas, prototypes, and code should come from everywhere. Nothing is so good that it can't be improved.

Sticky, sticky TiVo

Thomas Hawk discusses an article by Jeff Macke on the TiVo/Comcast deal. The key point is that TiVo customers are exceedingly loyal, reducing costly churn:

TiVo users ... are sticky -- apparently very sticky. For the fiercest TiVo owners they treat their TiVo like a religion ...

Churn costs cable operators and satellite companies money. Basically it costs them about $800 to get you as a customer in the first place.

The churn rate for DirecTV/TiVo users is .2% per month. "That's one full percentage point lower than the levels of churn for Comcast DVR customers ..." Reducing churn [by] 1% ... would save about $60 million per month.

[Comcast] recognize[s] that their own ... engineers have come nowhere close to matching TiVo's offering.

TiVo has an excellent user interface and novel features like recommendations. These features make the experience on any other PVR seem hollow and pathetic, making switching costs higher and churn lower.

Gannett on personalized news

In the press release for the Topix.net investment, Jack Williams (SVP, Gannett) said:

People want the news that's relevant to them - where they live, the business that they are in, the topics they care about.

Jack says people should get the news that is relevant to them. He thinks each person should see news on the topics and issues they care about. Jack says people want personalized news.

Seems like Gannett is joining Reuters and AP in believing personalized news is the future.

Topix.net acquired!

Katharine Seelye at the New York Times reports that Gannett, Knight-Ridder, and Tribune are each taking a 25% stake in Topix.net.

Congratulations to Rich Skrenta and the rest of the Topix.net team!

Topix.net focuses on fine-grained categorization of news by subject and location. They combined a broad news crawl with a clever, automated, classification system. Topix has hundreds of thousands of categories including narrow categories like nanotechnology and small little towns like Sequim, WA.

While it may be a little surprising to see newspapers making this deal, mainstream newspapers have been struggling mightily to deal with online news. This deal has substantial benefits.

Gannett, Knight-Ridder, and Tribune gain a well-differentiated news aggregator to compete against Yahoo News and Google News. The acquisition also gives them a powerful tool for enhancing their own news sites with related content and customization features using Topix's fine-grained subject categories. Finally, Topix has developed some expertise in targeting advertising to news -- something newspapers desperately need -- and Topix can transfer those skills to its new partners.

Great news, Rich. Congrats again!

Tony Gentile deserves credit for breaking this story on rumors earlier today. Also see the comments from Gary Price, Susan Mernit, John Battelle, Michael Bazeley, and Mike Masnick.

Update: Rich Skrenta (CEO, Topix.net) posts about the investment deal.

Update: There seems to be a fair bit of confusion about the valuation Topix.net received, in part created by Bambi Francisco first reporting only that "the funding was less than $5 million" but then updating her article to say:

The three newspapers bought 75 percent of the company for a market valuation below $100 million, but north of what venture capitalists valued Friendster, according to someone close to the deal. Friendster ... received a market valuation of $50-plus million.

My interpretation of this is that Topix.net was valued between $50-100M for the buyout and also received $5M in cash for expansion and future operations. Very impressive.

Update: A year and a half later, Topix.net raises another $15M in funding to add additional fuel its impressive growth. Very cool.

The network is just the beginning

Bill Burnham has an excellent post on the problems with a lot of social networking tools:

Social networking companies face several inherent challenges. The first is ... "input-output asymmetry" ... that in order to get utility out of a social networks, users must first invest significant amounts of time setting up and maintaining their networks .... They are neither autonomous nor self-healing.

Social networking is inherently an intermittent and dynamic activity. Without some kind of application to force the regular use and maintenance of such networks, pure play online social networks are destined to become as stale and appealing as two week old bread.

Bill's argument is against networking just for networking. For example, on Friendster and Orkut, there is no goal. You go there, list your friends, and then... what? Once you have your network built, what do you do with it?

Instead, Bill says, focus on an application. A social network should be a tool that helps accomplish some other goal. Building a network should be a means to an end, not the end itself.

See also my earlier post, "What has become of Orkut?" and another post by Bill, "Earth to Friendster: We have a problem".

AOL + Kayak = Pinpoint Travel

Chris Sherman reports that AOL took a minority stake in Kayak, a travel metasearch startup, and partnered with them to launch a new site, Pinpoint Travel.

Ask Jeeves to be acquired?

Geraldine Fabrikant at the New York Times reports that:

IAC/InterActiveCorp ... is close to an agreement to acquire Ask Jeeves Inc. ... for about $1.9 billion.

IAC/InterActive owns ... Expedia, Ticketmaster, Home Shopping Network, Match.com and CitySearch.

There's been a lot of deals lately. Ask Jeeves itself acquired Bloglines just a few weeks ago. Yahoo acquired Oddpost, Farechase, and Flickr. MSN acquired Lookout. Google acquired Keyhole and Picasa. Looksmart acquired Furl.net. Mamma acquired Copernic. Six Apart acquired LiveJournal.

But this would be a much larger and riskier deal. While there's a certain logic to combining Ask and IAC, the business literature shows that the vast majority of mergers of large companies either destroy value or fail to create value. Promises of "synergies" usually fall victim to the staggering complexity of aligning the organizations.

[via Steve Rubel]

Update: Charlene Li has a good list of the expected synergies.

Update: It's official. Here's the press release.

Paul Graham, startups, and VCs

The latest essay from ubergeek Paul Graham is titled "A Unified Theory of VC Suckage".

Inflammatory title, to be sure, but the essay makes several good points. Paul's argument is similar to many of the points made by Joel Spolsky (of Joel on Software fame) in his excellent post, "Fixing Venture Capital".

[via Andrew Chen]

Update: Looks like Paul Graham is putting his money where is mouth is. His new Summer Founders Program is a paid summer internship designed to build new startups. From their FAQ:

We suspect that students, and particularly undergrads, are undervalued ... Our hypothesis is that many quite young hackers could start viable startups if someone just gave them enough seed money to develop a prototype. The Summer Founders Program is an experiment to test this hypothesis.

[via Joho]

Update: Joel and Paul did not go unnoticed by the VCs out there. Fred Wilson takes a conciliatory tone with his post, "We suck less". But Jeff Nolan gets out the big guns and flames entrepreneurs, Joel Spolsky, and just about everyone else. Kaboom! All very amusing.

Update: Seven months later, Paul Graham posts about the mostly positive results of the first Summer Founders Program.