Improving Amazon?

I want to surface a new comment thread on improving online shopping from one of my old posts. I would enjoy hearing other thoughts on it, so please comment if you have anything to add.

Chris Zaharias asked:

Where you think Amazon still has opportunities to improve their game?

I said:

Looking at the bigger picture, I think it is hard to say that Amazon is anywhere close to done. The experience of shopping at Amazon is hardly effortless, full of discovery, or even all that pleasant.

Going to Amazon should be like walking into your favorite store, the nearest shelves piled high with things you like, everything you don't need fading into the background.

When you walk up to an item, everything you need to quickly evaluate it and decide whether to buy it should float to your attention.

Buying should be effortless, a couple clicks at most, with no unpleasant surprises (such as hidden shipping charges, delays, or belated out of stock e-mails).

Amazon has taken some steps toward that vision, but is a long way from there.

What do you think? How do you think online shopping be improved? Are there things Amazon should be doing that they are not?

Learning forgiving hashes

Googlers Shumeet Baluja and Michele Covell had a paper at IJCAI 2007, "Learning 'Forgiving' Hash Functions: Algorithms and Large Scale Tests" (PDF).

The paper describes finding similar songs using very short audio snippets from the songs, a potential step toward building a music recommendation system. The similarity algorithm used is described as a variant of locality-sensitive hashing.

First, an excerpt from the paper describing LSH:

The general idea is to partition the feature vectors into subvectors and to hash each point into separate hash tables... Neighbors can be [found by] ... each hash casting votes for the entries of its indexed bin, and retaining the candidates that receive some minimum number of votes.

What is interesting (and a bit odd) about this work is that they used neural networks to learn the hash functions for LSH:

Our goal is to create a hash function that also groups "similar" points in the same bin, where similar is defined by the task. We call this a forgiving hash function in that it forgives differences that are small.

We train a neural network to take as input an audio spectrogram and to output a bin location where similar audio spectrograms will be hashed.

A curious detail is that initializing the training data by picking output bins randomly worked poorly, so instead they gradually change the output bins over time, allowing them to drift together.

The primary difficulty in training arises in finding suitable target outputs for each network.

Every snippet of a song is labeled with the same target output. The target output for each song is assigned randomly.

The drawback of this randomized target assignment is that different songs that sound similar may have entirely different output representations (large Hamming distance between their target outputs). If we force the network to learn these artificial distinctions, we may hinder or entirely prevent the network from being able to correctly perform the mapping.

Instead of statically assigning the outputs, the target outputs shift throughout training ... We ... dynamically reassign the target outputs for each song to the [bin] ... that is closest to the network's response, aggregated over that song's snippets.

By letting the network adapt its outputs in this manner, the outputs across training examples can be effectively reordered to avoid forcing artificial distinctions .... Without reordering ... performance was barely above random.

I have not quite been able to make up my mind about whether this is clever or a hack. On the one hand, reinitializing the target outputs to eliminate biases introduced by random initialization seems like a clever idea, perhaps even one that might have broad applicability. On the other hand, it seems like their learning model has a problem in that it does not automatically learn to shift the outputs from their initial settings, and the reordering step seems like a hack to force it to do so.

In the end, I leave this paper confused. Is this a good approach? Or are there ways to solve this problem more directly?

Perhaps part of my confusion is my lack of understanding of what the authors are using for their similarity metric. It never appears to be explicitly stated. Is it that songs should be considered similar if the difference between their snippets is small? If so, is it clear that is what the NNet is learning?

Moreover, as much as I want to like the authors idea, the evaluation only compares their approach to LSH, not to other classifiers. If the goal is to minimize the differences between snippets in the same bin while also minimizing the number of bins used for snippets of any given song, are there better tools to use for that task?

Universal action and the future of the desktop

In a Google engEdu tech talk, "Quicksilver: Universal Access and Action", Nicholas Jitkoff provides some thought-provoking ideas on the future of the desktop.

Starting at 03:52 in the talk, Nicholas begins describing how he thinks the desktop should work, listing four categories of user goals: Search, Summon, Browse, and Act.

Search is a fast, comprehensive, and easy-to-use desktop search tool, such as Google Desktop. This may not sound new but, amazingly, this has only recently begun to be a common part of the desktop experience.

Summon is using desktop search for navigation. You know something exists, you just want to get back to it immediately.

Browse may sound close to what the desktop does now, but Nicholas seems to mean browse not as navigating a folder hierarchy but as finding objects related to or near other objects. For example, you might not remember the name of the specific song you want, but you might be able to remember the artist who wrote it; getting to the first allows you to recall the second.

Act is when you want to immediately do a task (e.g. play a music track) without any intervening steps such as opening an application.

Note the deemphasis of the traditional file hierarchy, the focus on objects, and the shift away from applications and toward actions on objects.

The desktop should seek to satisfy our goals immediately. We should not have to start to adjust the lighting in a photo album by navigating an hierarchical menu, locating an application that allows you to edit photos, waiting for the application to load, and then opening the files using the open menu in that application. We should just ask to adjust the lightening in a photo album.

The next few minutes of the talk further break down some of the constraints on the desktop metaphor. Nicholas advocates fast, universal access that ignores the boundaries of the machine, reaching out to the network to whatever data and code is needed to act. The focus should be on the task -- getting work done -- using whatever resources are necessary, requiring as little effort as possible.

The vision is fantastic and inspiring. However, while Quicksilver is an interesting example, from what I saw, it appears to be only a baby step toward these lofty goals. The learning and automation appears primitive, and the effort required to customize severe, which may make Quicksilver closer to a geek tool than a realization of the broader ambition.

Even so, Nicholas is offering intriguing thoughts on where the desktop should go. It is well worth listening.

More interviews on search in 2010

Gord Hotchkiss at Search Engine Land continues his interviews on the future of search with his post, "Search In The Year 2010: Part Two".

Again, I would recommend reading the whole thing, but I here will focus on the parts on personalization.

In particular, there are a few tidbits on personalized advertising in this round of interviews. Some excerpts:

Chris Sherman: ... As they get to know you and your preferences, you know... "I never click on that video ad," they’ll gradually stop showing you [those] ads ... and maybe increase the ads ... that you do click on.

Larry Cornett: ... The more they understand about what a specific user is looking for in their context, the more intelligent they can be about what they're actually offering ... By being more targeted it will add more value for the users and hopefully, be a better experience for them as well .... Do [users] really want to spend time in the context where they're seeing a lot of stuff that’s not targeted and not appropriate and might even be annoying or would they rather ... [see ads that] could be beneficial for them.

[Gord Hotchkiss:] Personalization of advertising will happen incrementally and the ability to target accurately will improve over time. For many users, it will be a mixed environment, with some very well targeted, relevant ads in some locations that don’t even look like advertising and the more typical forms of untargeted advertising we're more familiar with.

The impression I got from this is that personalized advertising is now seen as inevitable. Privacy concerns may make it appear incrementally, but most seem to agree that it will happen.

On a different topic, usability guru Jakob Nielsen used his time to promote NLP over personalization and pick on Amazon.com's recommendations yet again. Gord asked me to respond.

On the one hand, I agree with Jakob about the long-term promise of natural language techniques (though I think he may be underestimating the challenges and overestimating the likelihood of rapid progress there) and his criticism of inaccuracies in personalization and recommendations (and they are inaccurate, no doubt).

On the other hand, I think Jakob is using an absolute measure of the effectiveness of personalization where a relative measure is more appropriate. Specifically, the metric should not be how often does personalized content accurately reflect your interests; it should be how much better does personalized content predict your interests than whatever unpersonalized content you otherwise would have to put in the space.

That is a much lower bar. Bestsellers and other unpersonalized content tend to be very poor predictors of individual interest. By knowing even a little bit about you, it is easy to do better.

Is personalization ever going to be perfect? No, but it does not have to be. It just has to be more useful than the alternative. Personalized content only has to be marginally more interesting than unpersonalized content to be helpful.

See also Gord's first post in this series, "Search In The Year 2010", which has some more on personalized search, and my comments on that post.

For more on personalized advertising, please see also my posts "What to advertise when there is no commercial intent?" and "Is personalized advertising evil?"

The power of branding in web search

Some research out of Penn State, "The Effect of Brand Awareness on the Evaluation of Search Engine Results" (PDF) puts some hard numbers on the hurdles for Google's web search competitors.

The study showed participants Google search results on all queries, but switched around branding elements at the top and bottom of the page to label the results as from Yahoo, Microsoft Live Search, a startup called AI²RS, and Google.

From the paper:

Based on average relevance ratings, there was a 25% difference between the most highly rated search engine and the lowest, even though search engine results were identical in both content and presentation.

The 25% difference was between the results branded with the AI²RS startup and the results branded as Yahoo.

Curiously, Yahoo was rated substantially higher than Google, despite the fact that these were Google's search results. Yahoo has failed to gain web search market share, but, if you believe this study, brand weakness is not the reason why.

It is true that this study is small, just 32 participants and across 4 different queries. It would be nice to see a broader study that confirms these results.

Even so, it probably is safe to say that the strength of the Google and Yahoo brands (and Microsoft's ownership of the defaults in Internet Explorer) make it very difficult for any web search startup. As Rich Skrenta once said, "A conventional attack against Google's search product will fail ... A copy of their product with your brand has no pull."

See also a lighter Penn State Live article on the study, "Branding matters -- even when searching".

[Found via Barry Schwartz]

Missing an opportunity in shopping metasearch

There is an interesting tidbit at the beginning of a Search Engine Land post, "Kicking The Tires On Shopping Search, Part Two: The Independents".

Compiling data from Hitwise, they shows that Google, Microsoft, and Yahoo now have a meager 15% market share in shopping search combined, down from nearly 50% combined three years ago.

Considering that a substantial percentage of web search queries are shopping-related (see "A taxonomy of web search" (PDF)) and the ease of extracting advertising revenue and revenue sharing where there is such strong purchase intent, I would think that Google, Yahoo, and Microsoft would be pursuing shopping metasearch more aggressively.

See also part one of the Search Engine Land series, "Analyzing the Major Shopping Search Services", which focuses mostly on design and usability of the shopping sites offered by Google, Yahoo, and Microsoft.

See also my earlier posts, "R.I.P. Froogle?" and "What should Google do next?".

HITS, PageRank, and keeping it simple

A SIGIR 2007 paper out of Microsoft Research, "HITS on the Web: How does it Compare?" by Marc Najork, Hugo Zaragoza, and Michael Taylor is a large-scale study of several ranking algorithms using a substantial web crawl and data from the MSN query logs.

The authors appear to have expected the HITS algorithm to outperform the others in their tests, but found instead that a combination of BM25F and simple in-degree link analysis outperformed everything else. From the paper:

We were quite surprised to find that HITS, a query-dependent feature, is about as effective as web page in-degree, the most simpleminded query-independent link-based feature.

As expected, BM25F outperforms all link-based features by a large margin. The link-based features are divided into two groups, with a noticeable performance drop between the groups. The better-performing group consists of the features that are based on the number and/or quality of incoming links (in-degree, PageRank, and HITS authority scores); and the worse-performing group consists of the features that are based on the number and/or quality of outgoing links (outdegree and HITS hub scores).

The combination of BM25F with ... id in-degree consistently outperforms the combination of BM25F with PageRank or HITS authority scores, and can be computed much easier and faster.

PageRank performed poorly in their tests. However, their explanation of why struck me as unconvincing. From the paper:

The fact that in-degree features outperform PageRank under all measures is quite surprising. A possible explanation is that link-spammers have been targeting the published PageRank algorithm for many years, and that this has led to anomalies in the web graph that affect PageRank.

This begs the question of whether they picked the right PageRank algorithm. In particular, there are variants of PageRank that they could have used that appear less sensitive to spam and may have performed much better. Unfortunately, without results for those variants, it is hard to know whether the criticisms in this paper of naive PageRank are applicable to the algorithms evolved from PageRank used by search engines today.

Even so, the results of the study are interesting, both for the overview of several relevance ranking algorithms and the conclusions about their effectiveness. Particularly intriguing is the evidence that computationally expensive algorithms such as the query-dependent HITS algorithm seem to hold no advantage over much simpler techniques.

Update: Marc Najork, one of the authors of the paper, expands on the PageRank algorithm issue and the performance of HITS in the comments for this post.

Marissa Mayer at SES 2007

Elinor Mills at CNet summarizes Googler Marissa Mayer's keynote speech at SES 2007.

Some excerpts from the article, focusing on personalization:

For general Web search, personalization is the future, Mayer said.

Ten to 15 years from now search sites will understand more about searchers, where they are located and what their personal preferences are, she predicted.

Mayer said one of the most important data points for improving search relevance based on personalization is the previous query, although Web history and address books could also be helpful "signals" to the search engine.

It is important that the ads are personalized too, she said ... "My philosophy is that the ads and the search results should match ... For me, search and ads are almost the same."

See also detailed notes on Marissa's keynote from Tamar Weinberg.

For more on personalized search, you might be interested in at least a couple of my posts on that topic, including "Personalized Search Primer" and "Effectiveness of personalized search".

For more on personalized advertising, please also see my posts "What to advertise when there is no commercial intent?" and "Is personalized advertising evil?"

Collective search versus personalization

Eric Auchard at Reuters reports on Ask CEO Jim Lanzone's keynote at the SES 2007 conference.

Jim argued that personalization doesn't work and then tried to contrast it with something he called "collective search". Some excerpts:

Ask.com ... aims to tap the collective search habits of its 50 million users to improve the relevancy of Web search.

[Jim Lanzone] said attempts at automated personalization often fail in practice to give users what they want.

Instead, Web search can be improved by understanding the aggregate behavior of different types of users.

This collective approach means users stand to benefit from what users with similar interests have gleaned from previous searches.

"Collective search is something that Ask really believes in," Lanzone said, adding that personalizing what different users see is only a small piece of further improving search.

If I could quote from my favorite movie, "You keep using that word. I do not think it means what you think it means."

I admit the term personalization may be poorly defined these days, but it is hard for me to see the distinction between personalized search and changing the search results based on "what users with similar interests have gleaned from previous searches."

In fact, I would think that is the very definition of personalization. Personalization changes what people see based on their past behavior and the past behavior of others.

Perhaps the distinction here is that collective search may change results even for people who have no history? For example, popular search results may get a higher ranking or search results that appear to be related after analyzing what people click on may be handled differently?

Yet even that often still is referred to as personalization. For example, one of Amazon.com's most successful and useful personalization features is similarities ("Customers who bought X also bought"). That feature is targeted to a specific page, not to a user's history, but is still personalization.

Am I missing something more here? Is Jim suggesting collective search includes something that personalization does not? For example, does collective search include explicit sharing of search results across a social network (like Yahoo's struggling MyWeb)? Or something else?

For more details on Jim's interview, see also Tamar Weinberg's notes over at Search Engine Roundtable.

Personalization session at SES 2007

Barry Schwartz at Search Engine Roundtable posts notes from the SES 2007 session on "Personalization, User Data & Search".

Let me highlight this tidbit on the results of an eye tracking study:

The personalized [search results] ... doubled the performance (click throughs).

That reminds me of what Googler Marissa Mayer once said:

[Personalization is] one of the biggest relevance advances in the past few years.

Personalization doesn't affect all results, but when it does it makes results dramatically better.

Update: More details on the eyetracking study.

Update: Even more details on the eyetracking study.

Interviews on search in 2010

Gord Hotchkiss at Search Engine Land compiled an impressive group of interviews for his article, "Search In The Year 2010". He talked to usability guru Jakob Nielsen, Googler Marissa Mayer, Larry Cornett from Yahoo, Justin Osmer from Microsoft, Michael Ferguson from Ask, and search industry experts Chris Sherman, Greg Sterling, and Danny Sullivan.

From the introduction:

It was with a great deal of anticipation that I threw in front of them the same question: what will the search results page look like in 2010?

Here, aggregated and condensed, are their answers.

Go read the whole thing, but, as usual, I am going to focus here on the part on personalized search.

Chris Sherman, Jakob Nielsen, and Greg Sterling are skeptical about personalization. Personalization "is incredibly hard to do" because "language is so inherently ambiguous" and "you have to guess", not to mention "the so called creep factor."

Danny Sullivan, Justin Osmer, Michael Ferguson, and Larry Cornett are more optimistic. "We're getting close to a tipping point on personalization" where little or no effort ("a very low investment") yields "a lot of return" because searchers will "get a lot more out of [the] search experience" if the search engine knows more about them. Searchers "clicks and their footsteps will walk to the experience that is most delightful and easy for them to use," though we should be careful not to "ask the users to do work." "Google is onto something with their personalized search results," and "people are misunderstanding how sophisticated it can be."

Oddly, Marissa Mayer did not say much on personalized search this time around. In the past, she has said ([1] [2] [3]) that "[personalization is] one of the biggest relevance advances in the past few years", "personalized search is something that holds a lot of promise", and personalization is key for building "the search engine of the future."

Similar to what Esther Dyson said in her interview with Charlie Rose, Chris Sherman agrees with Gord Hotchkiss that "Google is holding a significant portion ... of their personalization algorithm in reserve" because there is "caution" that they might "alienate the searcher." Chris goes on to say, "They probably have tons of stuff that they're not showing us."

On personalization being incredibly hard to do, please also see my March 2005 post, "Personalization is hard. So what?"

Esther Dyson on personalization and Google

Esther Dyson has an interview on the Charlie Rose show that covered several topics, including health, space travel, search, personalization, and social networking.

At 34:22, she talks about personalization, personalized search, and behavioral targeting. Some quotes:

The big issue for Google is going to be personalization.

You sort of see them dancing around this issue of, well, I could do much better search for you if I knew [more about you].

They are very, very concerned about the privacy issue. They're terrified that this is going to be a problem for them.

It's pretty clear Google wants someone else to go first with personalized search to get people to be more relaxed.

Esther goes on to say that she expects smaller companies to take the lead on personalized search and advertising. The implication appears to be that, once these startups nicely warm up the public on personalization, Google will launch all the personalization features they have been holding back, using their big data and big clusters to dominate the field.

The rest of her interview is interesting, especially if you have a strong interest in health, but also just for the tidbits on other technology companies. For example, at 33:38, after a discussion of social networking, she says, "Yahoo should have become Facebook."

See also my May 2007 post, "Esther Dyson on the future of search".

[Charlie Rose talk found via Adario Strange]

Image search to solve hard problems

Alyosha Efros from CMU gave a fun Google Tech Talk, "Using Data to "Brute Force" Hard Problems in Vision and Graphics", with some clever ideas on using large image databases to solve hard problems.

The examples I enjoyed the most started about 20 minutes into the talk. The first, object insertion, looks at the problem of trying to add people or objects to a picture.

Rather than attempting to model the object to be inserted and then adjust the perspective, scale, and lighting, Alyosha suggests we change the problem to finding an appropriate object that already has the right perspective, scale, and lighting. That is, rather than take a specific picture of a man and try to adjust it, search a massive image database to find some picture of a man that is already properly adjusted.

This only works if you have a massive image database and good odds of finding a strong match, but we do now have massive image databases. These databases are only getting bigger with time, improving our odds of finding a good match and making this brute force approach appear even more promising.

Another example discussed in the talk, scene insertion, is a similar problem to object insertion, but at a larger scale. We are no longer just adding objects, but taking out whole chunks of pictures (e.g. construction equipment, roads, buildings, etc.) and creating a new picture by filling in the deleted data. In tools now available, this is done using texture fills, but that works poorly for large deletions.

Alyosha proposes a way of attacking the scene insertion problem where they search a large database for similar images. If they are trying to replace part of a picture that is mostly a beach scene, for example, they search our massive database to find similar beach scenes, then steal the missing chunk from the related scenes.

The talk is enjoyable and light with plenty of pictures of both good and bad examples of what happens when you try to apply this technique. Well worth watching.

See also my Oct 2006 post, "The advantages of big data and big clusters", where I quoted Googler Peter Norvig as saying, "Worry about the data first before you worry about the algorithm."

Effectiveness of personalized search

Zhicheng Dou, Ruihua Song, and Ji-Rong Wen from Microsoft Research had a great paper at WWW 2007, "A Large-scale Evaluation and Analysis of Personalization Search Strategies" (PDF).

There are three major conclusions in the paper: (1) Personalization only helps on some queries. (2) Both long-term and short-term history are important for personalization. (3) Profile-based techniques do not work as well as more fine-grained, click-based techniques.

On how personalization helps only on some queries, the key concept here is "click entropy", the amount of variation in the search results searchers click on.

For queries that are not ambiguous, the top result already may be ideal. If almost everyone clicks on that result, that query's click entropy is low. At least for these queries, there is little opportunity to improve the ranking using personalization (or, for that matter, using any other technique).

Thus, the authors conclude, "Personalized search has different effectiveness on different queries," and, "Click entropy can be used as a simple measurement on whether the query should be personalized."

On long and short-term history, the question here is whether personalization should focus on what you are doing right now (the last few searches) or your general interests (everything you have ever done).

The authors conclude that "the incorporation of long-term interest and short-term context can gain better performance than solely using either of them."

On profile-based versus click-based techniques, profile-based techniques record the high-level categories of interest for each searcher while click-based techniques modify the ranking of past and related clickthroughs.

The authors conclude that click-based techniques "work well", but profile-based "improve the search accuracy on some queries, but they also harm many queries" and are "not as stable as click-based".

These conclusions are not far from what I advocate on this blog. I pick on Google's personalized search for primarily being a profile-based technique focused on long-term history ([1] [2] [3]) and argue for a fine-grained, click-based approach. This MSR paper judges click-based techniques more effective, but also suggests that combining click-based and profile-based algorithms and using both long and short-term history may yield the best results.

See also my Sept 2006 post, "Potential of web search personalization", where I talk about a KDD 2006 paper out of Yahoo Research that comes to some of the same conclusions as this MSR paper.

Sep Kamvar on Read/WriteWeb

Richard MacManus at Read/WriteWeb has a short interview with Google personalization guru Sep Kamvar.

Using Wikipedia to disambiguate names

Silviu Cucerzan at Microsoft Research recently published a paper, "Large-Scale Named Entity Disambiguation Based on Wikipedia Data" (PDF), that is a great example of using the high-quality data available in Wikipedia to solve a difficult search problem, in this case, distinguishing between different meanings of the same name.

From the paper:

When ... scaling entity tracking to ... the Web, resolving semantic ambiguity becomes of central importance, as many surface forms turn out to be ambiguous.

For example, the surface form "Texas" is used to refer to more than twenty different named entities in Wikipedia.

In the context "former Texas quarterback James Street", Texas refers to the University of Texas at Austin; in the context "in 2000, Texas released a greatest hits album", Texas refers to the British pop band; in the context "Texas borders Oklahoma on the north", it refers to the U.S. state; while in the context "the characters in Texas include both real and fictional explorers", [it] ... refers to ... [a] novel.

Silviu cleverly uses the high-quality, semi-structured data available from Wikipedia for this task. In addition to pages describing different entities where contextual clues can be extracted (example), Wikipedia contains redirects for different surface forms of the same entity (example), list pages that categorize names (example), and disambiguation pages that show many of the different entities for a surface form (example).

Wikipedia contains much more than unstructured text. Exploiting the semi-structured data -- the redirect, list, and disambiguation pages -- gives this work its power.

For a quick overview on one of many ways this kind of named entity data could be applied, do not miss the screenshot in Figure 3 on page 6 of the paper. It shows a prototype that annotates a web page with pop-ups for the proper names to disambiguate the meaning.

As fun as this paper is, what really excites me is that this is one of many recent research projects that are cleverly using Wikipedia to attack challenging problems. There is little doubt that, deep in the Wikipedia pages, there is much buried treasure, if we can just figure out how to look.

Ambient Findability talk

Peter Morville gave a Google Tech Talk, "Ambient Findability and The Future of Search", that is a fun, light discussion about helping people find the information they need.

The slides from the talk are available (PDF), but they make little sense without Peter talking behind them. If you are in a rush and just want to skim the slides, make sure to see slide 23 with the definition of ambient findability and slide 27 on information overload.

There are tidbits on personalized search (which Peter seems skeptical about) and "keeping found things found" (aka re-finding) in the Q&A starting at 55:42 in the video.

Read/WriteWeb: Personalized Search Primer

I have a guest article on Richard MacManus' Read/WriteWeb, "Personalized Search Primer - And Google's Approach".

From the introduction:

Google has received much attention, not all of it positive, for its efforts to personalize search.

In this article, I will briefly describe personalized search, why Google and other search engines are trying to do personalized search, the approach Google is taking toward personalized search, and other approaches to personalized search.

Go read the whole article.

Google Search API for research

The Google Search API -- which, after deteriorating badly, was shut down about a year ago -- has been reborn in a version only available to researchers, the University Research Program for Google Search.

The promise of the API looks attractive at first. Google says:

The University Research Program for Google Search is designed to give university faculty and their research teams high-volume programmatic access to Google Search.

Our aim is to help bootstrap web research ... [Using] Google's search technology, you'll no longer have to operate your own crawl and indexing systems.

However, the limits on how the API can be used may be a problem. From the documentation:

Requests to the service MUST be throttled ... A time period of at least one second must be allowed between requests.

This makes some interesting types of research impossible using this API, anything that would need to fire off multiple queries quickly.

For example, let's say I am working on a technique for query expansion, so I want results not only for the search given, but also for tens of other related searchers, which I will then combine. With a one second delay between queries, my research prototype will take tens of seconds to respond, making it no longer interactive.

Nor can I try out some natural language analysis for question answering where I first get the results for the search given, then look at the results, then fire off dozens of additional queries to learn more about what I found in those results.

I cannot even do something that attempts to use conditional probabilities of finding two words together versus finding them apart on the Web as part of the analysis, since each of those requires two queries to the search engine and many of them might be required.

It is good that Google is making tools available to researchers, but they may have to go further than a throttled search API. As is, many researchers trying to work at large scale still will have to build their own crawls and indexes.

By the way, it is not entirely fair to pick on Google's search API here. Other search APIs -- including the Yahoo Search API, the Microsoft Live Search API, and the Alexa Web Search Platform -- either have a fee or similar throttling.

Google teasing too many lions?

Robert Cringely asks, "Is Google on crack?"

Inflammatory title, but Bob's point is that, while the telcos may be lumbering giants, taking on these powerful companies might just end with Google squashed underfoot:

Google made an unexpected reckless move in the wireless bandwidth market .... Google wants ... [to turn] what would have been yet another mobile phone system into a mobile Internet.

They don't know who they are messing with ... The wireless incumbents ... are mean and spiteful companies and WILL HAVE THEIR REVENGE.

The wireless carriers will spend whatever it takes to win ... because they don't want to change operational rules that have been very profitable for them over the years.

I'm all for tilting at windmills ... but Google has a lot at risk here and I think they are being foolish, even stupid.

What if Verizon, and AT&T, and Comcast, and half a dozen other huge broadband ISPs suddenly cut deals with some search company other than Google and your ISP-supplied browser and homepage no longer give such prominence to Google?

Yahoo ... fully supports Google's bold move, but you notice they didn't make it. Microsoft has been totally silent. Certainly Microsoft smells blood in the water and will be approaching all the outfits Google may have offended, trying to do exclusive search and ad deals with them.

Google seems to be biting off a lot these days. In addition to making Microsoft "hell-bent" to kill them, creating a spat with eBay, frightening the news media, angering the big movie and TV studios, and fighting Yahoo and the other search engines, now Google is threatening the telcos.

Google is teasing too many lions. These media and telco companies are massive. They are not beyond using market power to crush those they do not like.

It does not matter if Google's products are technically superior. It does not matter if Google's goals are noble. If all of these powers align against Google, Google will not survive.

Update: Apparently, Google likes to tease lions in multiple ways. The NYT writes, "Some believe another major goal of [Google's] phone project is to loosen the control of carriers over the software and services that are available on their networks."