BusinessWeek on Info-Overload

Steve Hamm at BusinessWeek writes about the future of search. He starts by slamming MSN's new search engine:

For proof that Internet search technology is still in its infancy, go to the Web site for Microsoft's new would-be Google killer. The company has invited people to try out a test version and suggest improvements. It could definitely use some. A search for Web pages related to the key words "Bill" and "Gates," for instance, starts off well with a couple of links to Gates' own Web pages on Microsoft's corporate site. But the results quickly devolve into a catch-all of random and even loony Gates-related material -- including a guide to hiring a William H. Gates III impersonator and a tongue-in-cheek spoof that tries to prove Gates is the devil.

That's a bummer. Microsoft has spent $100 million on the new technology -- its attempt to catch up with Google's great leap forward in Web search. Clearly, a lot more has to be done.

The problem, as Steve explains well, is that "there's still too much gravel among the nuggets of digital gold."

The burden is on the industry to turn the information tsunami into an unadulterated good. Until its engineers create much better ways of sorting through the sea of digital information that's flooding people's lives, consumers won't be able to get the most out of the Internet, and corporations won't receive full value for the nearly $8 trillion they have spent on info tech over the past decade. "We're just scratching the surface of what needs to be done," says Howard D. Wactlar, vice-provost for research computing at Carnegie Mellon University. "There's just so much information and so much of it isn't useful."

Microsoft acquires desktop search company

It's being widely reported that Microsoft has acquired a desktop search company called Lookout. Lookout's product adds a search bar to MS Outlook to allow easy searches of e-mail and files on the computer.

The Seattle PI predicted:

Microsoft's acquisition of Lookout Software reflects the fact that next-generation search engines are expected to focus on "unified search" -- encompassing not only data from the World Wide Web but also other sources of information, such as files stored on a hard drive. Microsoft has said it is designing its MSN Search program as a single system to search across a variety of data types, including PCs, databases and corporate intranets.

This acquisition is a bit unusual because of the size of the company. Lookout is a two-person company with one freeware product and only a few thousand users, according to the Seattle Times.

Update: Lookout apparently uses the Lucene open-source search search engine. This has lead some to question what technology was acquired by Microsoft in this deal. By the way, it's interesting to see that other users of Lucene include Furl, Nutch, and Seruku.

Security issues at Friendster

The latest Crypto-Gram newsletter points out a Wired article that documents some serious security holes in Friendster.

Moore has written several Unix shell scripts that run on-the-fly background checks on people who use wireless networks in his neighborhood. With the help of the popular network-traffic analysis utility Netcat, his script "sniffs all the traffic on the Wi-Fi network, greps for email addresses, and looks them up on Friendster." Then the script sends Moore an email that includes a link to the users' Friendster profiles, along with their pictures and login IDs.

At a time when it seems that nearly everyone has a Friendster account, Moore says, "You can do really creepy stuff. You can get the profiles on everyone in your local café, then see who their friends are, and just walk up to them and ask, 'Aren't you Tom's friend?'" More disturbing, Moore's toolkit allows him to get zip codes and last names, making it easier to track down the real-world addresses of his targets, thus opening up a whole new universe of creepiness. "You could do all sorts of mean things," he says.

[Another trick] mines for information about anyone who looks at his profile and clicks through to his Web site. "I get their user ID, email address, age, plus their full name. Neither their full name nor their email is ever supposed to be revealed."

Notified of the security holes, Friendster rep Lisa Kopp insists, "We have a policy that we are not being hacked." When I explain that, policy or no, they are being hacked, she says, "Security isn't a priority for us. We're mostly focused on making the site go faster."

"We have a policy that we are not being hacked." Wow. What more is there to say? Wow.

Default search in IE

The Google Toolbar now can override keyword search in the IE address bar, using Google instead of MSN Search.

Defaults have always been a big advantage to MSN Search. Windows and IE have dominant market share. Many novice PC users apparently don't even know what search they use; they just type keywords in address bar and search, so they use MSN Search by default.

But now, the Google toolbar makes it easy to change the IE defaults to use Google instead.

Newspapers online and offline

Adam Penenberg at Wired has a recent article that starts by talking about how the New York Times articles and archives are poorly represented in web search engines like Google. But the article later moves into a discussion of online news in general and what traditional newspapers should think about for their news websites.

One issue is that online readers are much more numerous but much less dedicated than print subscription readers.

The Times attracts 9 million unique visitors a month, while only about 1 million read the daily paper. On average, visitors spend about 43 minutes a month on the website, according to Nielsen/NetRatings, which gives them just enough time to sample a story or two that interests them. That's a fraction of the 28.2 minutes per day (3.4 times per week) a typical reader spends on the morning newspaper.

There are huge differences between the behavior of your online and offline readers. News websites should optimize for more casual and ephemeral online visits by taking advantage of the online format. Specifically:

The Times should customize its content so that readers could pick and choose which stories they want based on their own particular interests, rather than having to wade through the site's table of contents.

What is being suggested here is personalized news. Take advantage of the online media format. Customize each page to each reader's interests.

Because each reader spends so little time online (1.5 minutes/day online vs. 28.2 minutes/day offline), online readers generate much less revenue for newspapers ($11/user online vs. $900/user offline). By responding dynamically to the needs and interests of their customers, online news sites can increase the amount of time their readers spend on their site and increase advertising revenue.

Top online news sites

CyberJournalist.net posted the Nielsen/NetRatings of the "Top 20 Online Current Events & Global News" sites. CNN, Yahoo News, and MSNBC are dominant with Google News at about 30% the level of the market leaders.

Google vs. oFoto

Google acquired Picasa recently, signaling their entry into digital photo storage. Google's systems are designed to store large amounts of data across their cluster, so this business seems like a nice fit for them.

John Battelle speculates that travel will be next, but I have my doubts. The travel business is nasty, requiring interfacing with complicated legacy systems like the SABRE database. It's not clear to me that Google has any expertise or competitive advantage in this area.

Innovative recruiting from Google

A billboard on 101 near Palo Alto says nothing but:

{ First 10 digit prime in consecutive digits of e }.com

As described by Rupert Goodwins at ZDNet UK, solving this puzzle lead to another harder puzzle, then a page for jobs at Google.

Cute idea. Clearly it grabbed attention. I'm not sure the scheme will actually meet the goal of generating high quality resumes though. First, the selection criteria of solving these kinds of puzzles doesn't seem like a strong match with finding smart, talented people who will succeed at Google. Second, now that the final page is public knowledge, the filter, such as it was, is gone.

Nevertheless, you have to admire the creativity.

News.com Extra: Filtered by humans

CNet recently launched a general news site called News.com Extra. The tagline is "The Web filtered by humans, not bots," an amusing dig at news aggregators like Google News.

Of course, the tagline is absurd, implying equivalence where there is none. It's simply not possible to manually filter the hundreds of thousands of articles spidered daily by a news aggregator like Google News. Where an automated process can take advantage of the full breadth of news sources available, human editors will only have time to review and choose from a negligible fraction of the content available. The two are in no way equivalent products.

Not to mention that, if you want a news front page personalized to your interests, it can't be done using human editors.

Microsoft cuts benefits

Microsoft recently cut employee benefits and reactions from employees have been negative.

The Seattle Times quotes one person as saying, "Microsoft's benefits used to somewhat make up for what is a difficult place to work. Are we now going in the direction that it will be both difficult and unrewarding?" The article goes on to summarize the comments by saying, "The comments have one recurring theme: The cuts are leading to a drop in morale."

Cutting benefits can be dangerous. Typically, with any salary or benefit cut, your best employees (who have the best job prospects) leave at a disproportionate rate. It almost always has a negative impact on morale and productivity.

Moreover, benefits often are valued by employees at a level beyond the pure monetary value. One of the more interesting books I've read on employee compensation, Strategic Human Resources, makes this point:

Benefits and perks can also be particularly powerful symbols of gift exchange, moving the employment relationship from one with purely economic connotations to something more along the lines of a kin or friendship relationship. Salary, wages, and even bonus payments all have the connotation of an economic exchange in which each party should attempt to extract the best possible (narrowly selfish) deal. Some forms of benefits and perks are of an entirely different flavor and can cause the worker to respond with reciprocal gifts or by internalizing the welfare of the organization.

The psychological leverage associated with providing benefits is likely to depend on whether the employer is a pioneer in providing this perquisite or instead simply seen to be matching the competition.

It's interesting to note that Google offers a particularly exceptional benefits package.

Update: Well, look at that. Fourteen months later, BusinessWeek blames low morale and loss of key people, in part, on these benefit cuts.

The polarization of news

This week's Media Myopia column points to the increasing polarization of news. People tend to trust and read news sources that agree with their political beliefs. The article notes that:

Perceptions of "media credibility" - that is, whether people think a particular news outlet can be trusted - are now more driven by ideology and partisanship than at any point in nearly 20 years of surveys.

I would think that the best way to mitigate this issue would be to use a news aggregator like Google News or a personalized news aggregator like Findory News so you see your news from a large number of sources and viewpoints. But I was surprised to see the column cite the book Republic.com to argue against personalized news:

The Internet's ability to provide personalized news - to permit users to filter out those things they don't care about - [poses] a threat to democracy itself. Democracy depends in part on people's being exposed to information they would not necessarily have chosen for themselves. So, too, might the concept of gut rationality be endangered in a filtered world, where people see only what they want to see, hear only what they want to hear, read only what they want to read.

The author has a fundamental misunderstanding of the objective of personalized news. Personalized news does not seek to show a reader "only what they want to see." It does not pigeonhole a reader to one particular viewpoint. Rather, it helps the reader discover sources and articles that otherwise might have been missed.

While you may be a regular reader of the New York Times, Wall Street Journal, a news website, or your local paper, even the biggest news junkie only reads a few sources every day. Using Findory News, you now have a window into thousands of news sources you don't normally see, different viewpoints, different perceptions, different analyses. Rather than limiting your perspective, personalized news broadens your view, helping you find articles you never would have discovered on your own.

AltaVista code stolen by MSN Search employee?

Todd Bishop at the Seattle PI has the scoop.

While the prosecutor on the case said that the allegations currently "do not pertain to Microsoft," it's not known if any AltaVista source code ended up in the new MSN Search engine. AltaVista, by the way, is now owned by Yahoo.

The search wars keep getting more and more interesting.

NYT review of the new MSN Search

The NYT State of the Art column has a lukewarm review of MSN Search's UI redesign and their new search engine. Some interesting snippets, starting with the MSN Search redesign:

MSN [Search] still relies on search technology licensed from Yahoo. For many searches, the results are pretty much the same at Google. But do enough searches, spend enough months, and Google gradually re-earns its reputation for superior accuracy. The new MSN Search looks like Google but doesn't work like Google.

On paid placement:

Unfortunately, Microsoft calls the separation of advertising an experiment, not a permanent change in policy. It seems to be trying on honesty in the mirror to see if people will find it attractive, rather than realizing that running a principled business is the way to win customers' trust. In short, "MSN will continue to evaluate the potential of paid inclusion to improve relevancy."

And, finally, a harsh review of Microsoft's early attempt to build a new search engine:

You can try out Microsoft's extremely early version of this new search algorithm at techpreview.search.msn.com, but don't get your hopes up. First, the tech preview is very, very slow. Second, it should be more discerning; if you search for "motorized draperies," 13 of the first 15 results all come from the same company's Web site. Third, kill the sense of humor: the first result in a search for "1963 Oscar winners" is "1998 Oscar winners." (The actual 1963 winners don't even appear on the first page of 15 links.) In fact, on many searches that Google and Yahoo aced - "Richard Nixon's dog," "fertility over 40" and "Britney wedding photos," for example - the tech preview's first results page comes up empty-handed.

See also my earlier comments on the new MSN Search.

GMail having scaling trouble?

Some reports ([1] [2] [3] [4]) of trouble using GMail lately. Perhaps they've been a little too quick to give out those invites?

Slash(dot) and Burn

A paper by Cliff Lampe and Paul Resnick called "Slash(dot) and Burn: Distributed Moderation in a Large Online Conversation Space" has a fascinating analysis of data from Slashdot. Well worth reading. I particularly liked the table computing correlations between characteristics of a comment and the final moderation score of that comment.

Near the end of the paper, they propose some intriguing changes to Slashdot to address their primary concern, timeliness of the moderations:

Alternative designs might cause treasures to be discovered more quickly and consistently, at the expense of a little more moderator effort. For example, there could be a special moderator's view of a conversation. It would hide comments below certain thresholds, as with the view presented to other readers. But comments the system had flagged as needing additional moderator attention would not be hidden. Recently posted comments and those with recent moderation would be flagged. Once a flagged comment had been presented to enough moderators, the system would infer from the lack of any explicit moderator action that the item was correctly classified and stop highlighting it for future moderators. All comments would reach their final score much faster, and the problems of uncorrected moderation errors and buried treasures would be reduced significantly.

VP at Yahoo on personalized search

David Mandelbrot, Yahoo’s vice president of search content, recently had this to say about long-term plans for Yahoo Search:

Down the road we are focused on personalized search. We have a team of engineers working on discovering user intentions when they do a search. We’re looking at folders and clustering and other personalization techniques. Relevancy and freshness are huge priorities for us and our technology does a much better job in those areas as well as in comprehensiveness.

Blogory RSS feeds

Findory Blogory just launched a new and unusual feature: personalized, aggregated, and adaptive RSS feeds.

The RSS feed is personalized, so every reader gets a unique feed with articles that match his or her interests. The feed aggregates weblog articles from thousands of other weblogs, combining all the weblogs into one RSS stream and helping readers find and discover new articles and new weblogs. And Blogory's feed is adaptive, learning your interests as you read articles from the feed and changing the RSS feed to more closely match your interests.

To my knowledge, this is the first personalized, adaptive RSS feed. There's nothing else like it out there.

Many people might want to add it to their weblog reader as an easy way to read weblogs. For people who are overwhelmed with tens or hundreds of RSS feeds listed in their reader, Blogory might be a way of cutting through the glut, helping them find what they need each day without having to search manually through a long list of feeds. For others, Blogory may be a way of discovering interesting new weblogs.

Blogory in UK Guardian

Findory Blogory got a brief mention in the UK Guardian today. Interesting that the author of the article is Tara Calishain, lead author of the book Google Hacks and editor of the popular weblog ResearchBuzz.

Text-only option from Google cache

Google Blogscoped noticed that Google now allows you to see a text-only version of cached pages. For example, try the normal and text-only cached versions of AnandTech. If a site is down or very slow, this is a convenient feature.

This reminds me of a prototype that I wrote a few years ago that allowed people to browse the web entirely using the Google cache. All URLs were rewritten on the fly to point back to the Google cache, so, once you started browsing the web using the tool, you stayed on Google cache for all your web browsing. It was a cute idea and worked fairly well, but I didn't pursue it beyond the prototype.

While you're at Google Blogscoped, check out their little EgoBot toy. I love the answer to "What is... personalization?"

I'm curious how EgoBot is implemented. Again, a couple years ago, I had a prototype called "The Oracle of Google" that took a question, executed a query against Google, and them did some simple natural language processing to try to extract likely answers from the blurbs in the Google results. It worked surprisingly well. I'm guessing this does something similar.

If you're interested in question answering using search, there's some research work that might be useful to you. One of my favorite papers is by Cwok, Weld, and Etzioni. It's called "Scaling Question Answering to the Web" and it describes a more principled approach to this problem, leveraging WordNet and some other publicly available tools for the natural language processing. They had problems with scalability -- the natural language processing was quite expensive -- but it's a fascinating approach to the problem.

Update: Microsoft Research has been doing a fair amount of work on question answering. Here's a paper I found particularly interesting.

New MSN Search launched

It's being widely reported (Yahoo News, FT, and more) that MSN has launched its new search engine.

The best write-up I've seen is by Danny Sullivan at SearchEngineWatch. John Battelle also has a write-up focusing on paid inclusion.

If you want to try MSN's new search, it's available in the MSN Sandbox. MSN Search isn't expected to be powered by this search engine until into 2005. As Danny Sullivan says:

The new Microsoft search engine is NOT -- NOT NOT NOT -- being used at MSN Search. It can be confusing, because along with the search technology announcement, Microsoft has also announced a new look and feel for its MSN Search site. Despite these cosmetic changes, under the hood, MSN Search itself still beats with a Yahoo heart.

Reviews of MSN's new effort so far have been lukewarm at best. Danny Sullivan had this to say:

The new search engine also leaves me with a "more of the same" feeling. It doesn't take search results anything beyond what Yahoo, Google or Ask Jeeves already do, and given their maturity, do better. In fact, a fast run of the tested queries through Gigablast -- a one man effort by Matt Wells -- makes you think MSN still needs to catch up to even that service.

But, to be fair, it's early. MSN Search may be an also-ran right now, but they've got a massive development effort behind them and intend to try to differentiate themselves soon. And it appears that personalized search will be their primary means of differentiation. From the MSN's message about their commitment to search:

A Personalized Experience. Your Search service should learn from you. What you like, what you read, where you live. Search should deliver results that are more personal and relevant to you.