Google and the long tail

Dan Gillmor talks about Google and how its advertising model addresses the long tail of the advertising market.

See also the NYT on the "publishing revolution" from AdSense and my earlier comments on Google's advertising model.

Google wants to keep it simple

Xeni Jardin posted a "rough transcript" of a talk by Eric Schmidt (CEO of Google) today. An excerpt:

The average person does not want to debug their computer. We prefer instead the idea of a person typing something in and Google -- or someone else -- figuring things out for you. But very few things are organized around that principle of simplicity; we love and appreciate the complexity in technology but people using the internet really don't want that. When you see an ease of use breakthrough, it's such a wonderful thing.

Things should be simple. It should just work.

As obvious as this sounds, simplicity is not the focus of Google's competitors. For example, My Yahoo is focused on customization, an approach that requires a lot of work from its users. MSN Search leaked a design that seems headed toward more knobs to twiddle and more complexity. And even though Google knocked off AltaVista with a simple keyword search that just worked better, smaller competitors these days such as A9 and Snap.com seem to be focusing on complicated interfaces with lots of little buttons to push.

But Google gets it. The computer should help you, not require work from you.

Update: The Economist is running an article, "Keep it simple", that claims, "The next big thing in technology... [is] the conquest of complexity."

Yahoo home page redesign

Yahoo launched a redesign of their home page. Still cluttered, still poorly prioritized.

See also Jerry Yang's recent comments on the Yahoo home page clutter.

[via Google Blogoscoped]

Update: Terry Semel (CEO of Yahoo) says personalization is coming soon to the Yahoo home page. Here's my advice to Terry.

Delicious Library.com

Niall Kennedy posts about Delicious Library, a clever and gorgeous Mac OSX application that keeps track of the books, music, and video titles you own. It's a fantastic use of the Amazon Web Services API.

Udi Manber (former "Chief Algorithms Officer" at Amazon.com, now CEO of A9) motivated Amazon's Search Inside the Book feature by saying:

"Why not have users take pictures of their bookshelf? We could scan the images, extract the titles, and then let them search the entire text of the books they own."

Delicious Library is a promising step toward that goal.

Interesting how Amazon's extensive web services API is spurring innovation outside of Amazon.

Update: John Siracusa has a detailed review of Delicious Library.

Amazon.com: Truly Data Driven

Another interesting talk at University of Washington, this time by Dennis Lee from Amazon.com.

Amazon.com has grown from a small bookseller to a place where you can literally find discover and buy almost anything. As we have grown, we have learned key lessons by leveraging the data that our customers provide to us: where they click, what they search for, and what they buy. I will go over the evolution of our software platform from the early days to today. I will then highlight several areas across the company where we use data in interesting ways that reveal very rich yet unintuitive behavior.

The talk will be broadcast live on the internet at 3:30pm (PDT) on Oct 28 and archived a few days afterward.

Update: The archive for this talk is finally available. I watched the first part of it live, but then got booted off because of heavy traffic. Guess it was a popular talk. I'm looking forward to watching the rest of it.

Update: The talk was surprisingly light, more of an introduction to Amazon and its features than a technical talk about data mining. The section on A/B testing (the tests used to determine the impact of design changes and new features) was good and might be particularly interesting to many. There also was a quick overview of some personalization features and supply chain optimizations.

Froogle adds merchant reviews

Google's metashopping search, Froogle, adds merchant reviews. The reviews seem to be spidered off other sites. For example, merchant reviews for Buy.com were from Bizrate, Shopping.com, and PriceGrabber.

There doesn't appear to be any way to enter your own review. That's too bad. Building a catalog of authoritative and trusted merchant and product reviews would make Froogle more compelling and would be a great differentiator. One of eBay's strengths is in being able to see the reputation of sellers, a type of merchant review. One of Amazon's advantages over other online merchants is the quality and depth of its product reviews. If Froogle is serious about helping people shop, it needs to help people find authoritative and reliable reviews.

Speaking of merchant reviews, there's still no merchant reviews on Google Local. Yahoo Local does have merchant reviews.

Merchant reviews will be a key feature for local search. It's hard to get authoritative, reliable reviews of local merchants. Services like Citysearch, ePinions, or Zagat attempt to provide reviews, but coverage is spotty and reliability uncertain. A well implemented local search with merchant reviews could change this space forever.

Google acquires Keyhole

Google acquires Keyhole, a company that produces some nifty map visualization software.

[via Google Blog and others [1] [2] [3]]

Update: John Battelle talks with John Hanke (CEO of Keyhole):

Keyhole is in the Holy Crap That's A Lot of Data business - that alone is reason for Google to be interested. Their database stands at 12 terabytes and growing.

Hanke showed an application, which he called geoblogging, which allows folks to fly around Keyhole's data and annotate various things they see ... [and] others can view and comment upon.

Very interesting.

Google cheat sheet

This Google Cheat Sheet in the Google Help is actually kind of interesting. I didn't know about the synonym search (e.g. "~auto loan" will allow auto to match car, truck, etc.) and the wildcards (e.g. "red * blue" will match occurrences of red and blue with one word in between).

[via Google Blog]

Finding mirrored hosts

Here's an interesting technical problem. You work at a major search engine crawling billions of pages. How do you identify that two sites are copies of each other?

This problem turns out to be quite important to relevance rank. If two sites are copies of each other, they'll tend to distort PageRank (and similar link-based relevance ranks). The obvious solution, computing pairwise similarity of all documents, is completely impractical on a dataset of this size.

This question also seems to be a popular in interviews at Google. When I talked to them over a year ago, three different people asked me how I'd solve this problem. At the time, I said I'd use a content-based technique that, briefly, would compute several signatures for samples of the text of the documents and then look for matches for those signatures.

Turns out our good friend Jeff Dean co-authored a paper (PDF) on this very topic. The paper analyzes the performance of several techniques for detecting mirrors, from simple approaches like the similar IP address or hostname to more complicated and quite clever analysis of the link structure of sites. The paper concludes that a content-based approach (called "shingles" in the paper) works well but that a combination of several approaches works best.

The paper is accessible. Worth a peek if you have any interest in the topic.

Peek at the future of MSN Search

With a little sleuthing, Gary Price discovered what appears to be experimental prototypes for the new MSN Search.

Most curious is the use of sliders to modify the search results. Google Personalized Search has a similar slider feature, but I thought it was widely viewed as ineffective. Interesting to see MSN experimenting with it.

Update: Andrew Goodman thinks sliders could be big, giving users the ability to customize relevance rank as they wish.

I suspect sliders would be of interest only to power users, the kind of people who already use advanced search all the time. Most users want search to "just work". This kind of control is just a novelty, not something most would actually bother to use.

This discussion reminds me a bit of the debate between Udi Manber and Louis Monier at Web 2.0 on whether users want more powerful tools or just want the right thing to happen.

The search war hits the desktop

The BBC has a good overview of the current state of the desktop search and the impact of desktop search in the search war. Worth reading.

[via Findory News]

Do people know what they want?

Jon Udell points to a talk by Malcolm Gladwell on the design of the Aeron chair and the "instability of preferences":

[Gladwell] draws several conclusions. One is that preferences are highly unstable. Another is that, when you ask people to explain what they want, their preferences tend to shift toward the conservative, familiar, and easy to explain.

This is a major problem with customization. When you rely on people telling you want they want, they often don't really know.

And it's even worse than that. If you ask people to tell you what they want, you're throwing up a hurdle. You're requiring work. The vast majority of people won't even bother to tell you. For example, on a website like My Yahoo, most people just use the default, uncustomized My Yahoo page.

In the few cases where they do bother, they'll often provide an incomplete or inaccurate description of their preferences. If you rely solely on this data, you'll get their preferences wrong.

And in the very few cases where they provide a complete and accurate description of their preferences, they'll often fail to maintain it. The preferences will become increasingly stale and inaccurate over time.

Had the Aeron designers listened to what people said their preferences were, they never would have produced this chair. But, at the same time people were saying the Aeron chair was ugly, sales were skyrocketing.

How do you know what people really want? Watch what they do and learn from their behavior.

Joel on Google

Joel Spolsky of Joel on Software talks about Google in an interview on Microsoft Watch:

I think Google is orthogonal to Microsoft. Microsoft will be chasing their tail lights ... Google is saying the Internet is the computer — it's the Internet computer in the sky. And Microsoft's not getting it.

But one of the mistakes Google is making is applications like Gmail are great on the Google platform. But if Google was really paying attention, they'd say we have to have outside developers writing applications for Google. There should be 27 different e-mail systems using the Google infrastructure. And then they can become Microsoft. It will be interesting to see if Google will wake up and open themselves to developers. The (current) Google API is incredibly narrow and not open.

Great point on the Google API. If Google doesn't expand its API more soon, there's an opportunity here for Yahoo. But Amazon seems to get it. They've recently opened their API further and released an Alexa API.

Joel also speculates a bit on a Google browser:

I quoted a Microsoft guy (and Longhorn Avalon team member) named Joe Beda. I quoted him saying "Microsoft is making a big bet on the rich client." And now he works at Google with Adam Bosworth. I'm sure what they're doing is a new browser. It's the IE (Internet Explorer) team reconstructed inside Google.

[Thanks, Robert Scoble, for pointing out the interview]

Update: Hate to contradict Joel, but Eric Schmidt (CEO of Google) just said, "We are not building a browser."

Google's Kirkland office

In Jeff Dean's talk today at University of Washington, he officially announced that Google is opening an office in Kirkland, WA. The office is right outside of Seattle and, more importantly, right near Microsoft.

Clearly, this move is intended to steal top talent from the Redmond giant. Especially after recently cutting benefits, Microsoft may have difficulty retaining key people in MS Research and MSN Search.

Amazon, located in Seattle, should also be concerned. Amusing, in some ways, since Amazon just opened an office in Google's backyard. Turnabout is fair play.

The search war is heating up!

Update: Let me clarify. What's new here is that this is a software development office in Kirkland. Yes, Google has had a sales team up in the Seattle area for a while. But Jeff said Google is opening a branch office for software engineers here in the Seattle area.

Update: Coverage in the Seattle Times.

Update: Joe Beda talks about the move to the Google Kirkland office. There's apparently a housewarming party for the new office on Nov 18.

Update: Gary Price has details on the Google office, including the exact location and maximum capacity. [via InsideGoogle]

More on Google behind the scenes

Jeff Dean from Google gave a great talk today at University of Washington.

Much of the early part of the talk focused on their use of commodity hardware and their cluster, much of which is covered in the Google Cluster Architecture and Google File System papers.

Jeff spent some time talking about MapReduce -- a custom programming model used at Google for rapid develop of robust parallel applications -- and pointed to his upcoming MapReduce paper at OSDI.

He also demoed Google's new word clustering work, which can find related words given a word or phrase. For example, for "rolling hash", it found words related to pot, but for "rolling hash function", it came up with "MD5" and other one-way hash functions. It was also able to find synonyms like "cuisine" for "cooking", though non-synonyms but still related words like "food" also showed up high in the clusters. Jeff said Google will use this data to improve relevance rank and help find search results that are clearly relevant to your search query but don't exactly match your search terms. This demo looked similar to what I heard of Peter Norvig's talk at Web 2.0.

This word clustering is great example of what you can do with massive amounts of data and processing power. We did a lot of similar things at Amazon.com.

It's not how big it is, it's what you do with it

Nathan Weinberg posts on Yahoo's announcement of 1B images in their image search, saying:

These numbers are meaningless, and are simply blows in the search engine war, far too reminiscent of the mid-90s war. Google won the war by proving it was relevancy that mattered, not numbers. Yahoo is doing a good job fighting Google, but this fight is just on the wrong turf.

Exactly right. It's all about relevance.

You want to help people find what they need. The only way increasing index size would do that is if there's many cases where the index is missing an item that should be at the top of the search results. More likely is that most useful items are already in the index but just aren't being surfaced. In that case, the focus should be on relevance rank.

Update: Gary Price notes that "publicly announced pages totals can be useful but are often nothing more than a marketing tool" and that "relevant results are what counts."

Update: Jeremy taunts Google for having an out-of-date image index and boasts about Yahoo's 1B images.

Google behind the scenes talk

Jeff Dean from Google is giving a talk at University of Washington covering the Google File System and other nifty tidbits. The talk will be broadcast live over the internet at 3:30pm (PDT) today.

Update: If you missed it live, the talk is now available for download.

Diversity and automation in online news

Mark Glaser at OJR has a great article on deep linking, diversity, and automation on news sites.

There have been some tremendous changes recently. BBC is linking to related articles on other news sites. The New York Times is opening more of its archives. And even the Wall Street Journal is experimenting with providing access to more of its content.

Richard Deverell, head of BBC News Interactive, credits Google News with forcing the change:

"I think Google News has been a shot across the bow of all news originators, making us say 'hold on, there's a different way of doing this.' It's very easy to flip between different sources of news. We either try to reverse that trend, which is likely futile, or we facilitate it, and I'm keen that we take the latter route."

Deverell later talks about the advantages of diversity:

"People do not trust individual sources, no brand is trusted completely -- those days are over," he said. "And people value a range of perspectives. So, for instance, with a political story, we try to give a very impartial account of it, but then the left-wing press will give their perspective and the right-wing press will give a different perspective."

And the advantages of automation:

The one striking difference between News.com Extra and BBC Newstracker is that the former is created by human editors and the latter is largely a weighted algorithm similar to Google News. Deverell says having editor picks is a nice idea but is just too labor intensive, while the augmented Moreover feeds are "fantastically automated and therefore cheap."

[Thanks, JD Lasica, for the link to the OJR article]

Update: For more on the benefits of automation and a diverse pool of news sources, see my earlier post, Humans vs. Robots.

Google "Life Search"

BBspot has a hilarious satire piece on Google "Life Search":

Simply type in what you are looking for and Google Life Search will quickly locate that item. For example if I enter 'car keys' Life Search responds with the result 'In your pocket', and there they are right where it said! There is also a cached version to show you where your keys were."

Several regular contributors to the tech site Slashdot posted reports of suspected problems in the new search algorithm saying they entered the term "life" and were presented with a "Result Not Found" error. They were presented, however, with several ads for internet dating services and online gaming sites.

"This is the end of the personal privacy. Google now knows everything you read on the web, they search your e-mail and now they know about the dope hiding in your sock drawer."

Very funny, and a perfect followup to my previous post, Google Memex.

[via Andy Beal]

Google Memex

In a BBC article on Google Desktop Search and personalization, some interesting quotes from Marissa Mayer (Director at Google):

"We think of this as the photographic memory of your computer."

"If there's anything you once saw on your computer screen, we think you should be able to find it again quickly."

This is more than just desktop search. This is clearly inspired by Memex, the "memory extender" described half a century ago. It also sounds similar to the goals of "Stuff I've Seen", a research project at Microsoft. Everything you've ever seen on your computer is easy available again.

Others (A9, Seruku, Ask, My Yahoo Search, Microsoft) have a similar vision, but this is the first time I've seen it from Google. Because of privacy concerns, Google has been resistant to maintaining this type of data in the past, but those concerns seem to be gone now.