Experiments and performance at Google and Microsoft

Despite frequently appearing together at conferences, it is fairly rare to see public debate on technology and technique between people from Google and Microsoft. A recent talk on A/B testing at Seattle Tech Startups is a fun exception.

In the barely viewable video of the talk, the action starts at the Q&A around 1:28:00. The presenters of the two talks, Googler Sandra Cheng and Microsoft's Ronny Kohavi, aggressively debate the importance of performance when running weblabs, with others chiming in as well. Oddly, it appears to be Microsoft, not Google, arguing for faster performance.

Making this even more amusing is that both Sandra and Ronny cut their experimenting teeth at Amazon.com. Sandra Cheng now is the product manager in charge of Google Website Optimizer. Ronny Kohavi now runs the experimentation team at Microsoft. Amazon is Experimentation U, it seems.

By the way, if you have not seen it, the paper "Online Experimentation at Microsoft" (PDF) that was presented at a workshop at KDD 2009 has great tales of experimentation woe at the Redmond giant. Section 7 on "Cultural Challenges" particularly is worth a read.

Google AdWords now personalized

It has been a long time coming, but Google finally started personalizing their AdWords search advertising to the past behavior of searchers:

When determining which ads to show on a Google search result page, the AdWords system evaluates some of the user's previous queries during their search session as well as the current search query. If the system detects a relationship, it will show ads related to these other queries, too.

It works by generating similar terms for each search query based on the content of the current query and, if deemed relevant, the previous queries in a user's search session.

There have been hints of this coming for some time. Last year, there were suggestions that this feature was being A/B tested. Earlier, Google did a milder form of personalized ad targeting if the immediately previous query could be found in the referrer. Now, they finally have launched real personalized advertising using search history out to everyone.

For more on Google's personalized advertising efforts, you might be interested in my earlier post, "Google launches personalized advertising", on the interest-based behavioral targeted advertising Google recently launched for AdSense.

Please see also my July 2007 post, "What to advertise when there is no commercial intent?"

Rapid releases and rethinking software engineering

I have a new post up at blog@CACM, "Frequent releases change software engineering", on why software companies should consider deploying software much more frequently than they usually do.

Here is an excerpt, the last two paragraphs, as a teaser:

Frequent releases are desirable because of the changes it forces in software engineering. It discourages risky, expensive, large projects. It encourages experimentation, innovation, and rapid iteration. It reduces the cost of failure while also minimizing the risk of failure. It is a better way to build software.

The constraints on software deployment have changed. Our old assumptions on the cost, consistency, and speed of software deployments no longer hold. It is time to rethink how we do software engineering.

This CACM article expands on some of the discussion in an earlier post, "The culture at Netflix", on this blog and in the comments to that post. By the way, if you have not yet seen Reed Hastings' slides on the culture at Netflix, they are worth a look.

The culture at Netflix

Netflix CEO Reed Hastings has a very interesting presentation, "Our Freedom & Responsibility Culture", with some thought-provoking ideas on how to run a company.

Some excerpts:

Imagine if every person [you worked with] is someone you respect and learn from ... In creative work, the best are x10 better than the average ... [A] great workplace [is made] of stunning colleagues.

Responsible people thrive on freedom and are worthy of freedom ... [They are] self-motivating, [pick] up the trash lying on the floor, [and behave] like an owner ... Our model is to increase employee freedom as we grow rather than limit it ... Avoid chaos as you grow with ever more high performance people, not with rules.

Pay at the top of the market is core to high performance culture. One outstanding employee gets more done and costs less than two adequate employees ... We pay at the top of the market ... Give people big salaries ... no bonuses ... no stock options ... [and a] great health plan ... [Everyone feels] they are getting paid well relative to their other options ... Nearly all ex-employees will take a step down in comp for their next job.

We try to get rid of rules when we can .... The Netflix vacation tracking policy [is that] there is no policy or tracking. There also is no clothing policy at Netflix, but no one has come to work naked lately ... Netflix policy for expensing ... [is] five words long ... "act in Netflix's best interest" ... You don't need detailed policies for everything.

Reed also makes a great point about how to organize large companies, saying he prefers to align groups on goals and strategies while minimizing meetings over tactics. He contrasts this with "tightly-coupled monoliths" where everything is inefficiently controlled (usually from the top down) and "independent silos" where groups (e.g. engineering and marketing) work so independently that "alienation and suspicion" creep in.

There are some suggestions I disagree with. First, I think Reed's claim that Netflix should fire with "generous severance" people who managers would not "fight hard to keep at Netflix" if they were to threaten to leave conflicts with Reed's later advice that managers should not blame someone who "does something dumb" but rather ask themselves what "context [the manager] failed to set." Personally, when someone I manage is not doing well, I blame myself, not them, and I think Reed should have emphasized finding people the right challenge rather than suggesting just giving them the boot.

Second, I think Reed's advice to push new software to the website every two weeks is not nearly frequent enough -- I prefer at least daily -- and I also see this as at odds with his later claim that he wants "rapid innovation", "excellent execution", and "to be big and fast and flexible".

But, overall, a great presentation with excellent food for thought. It is a must-read for anyone thinking about how to use organizational culture to help manage a company, from little startups to bloated corporate empires.

Please see also my old 2006 post, "Management and incentives at Google", that discusses Google's corporate culture.

[Netflix slides found via Ruben Ortega, TechCrunch, and Hacking Netflix]

Update: Scott Berkun has some good thoughts on the slide deck, including nice references to Zappos' "pay to quit" idea and the "Lefferts law of management".

Rapid innovation using online experiments

Erik Brynjolfsson and Michael Schrage at MIT Sloan Management Review have an interesting take on the value of A/B tests in their article, "The New, Faster Face of Innovation".

Some excerpts:

Technology is transforming innovation at its core, allowing companies to test new ideas at speeds -- and prices -- that were unimaginable even a decade ago. They can stick features on Web sites and tell within hours how customers respond. They can see results from in-store promotions, or efforts to boost process productivity, almost as quickly.

The result? Innovation initiatives that used to take months and megabucks to coordinate and launch can often be started in seconds for cents.

That makes innovation, the lifeblood of growth, more efficient and cheaper. Companies are able to get a much better idea of how their customers behave and what they want ... Companies will also be willing to try new things, because the price of failure is so much lower.

The article goes on to discuss Google, Wal-mart, and Amazon as examples and talk about the cultural changes necessary (such as switching to a bottom-up, data-driven organization and reducing management control) for rapid experimentation and innovation.

I am briefly quoted in the article, making the point that even failed experiments have value because failures teach us about what paths might lead to success.

Can we make make all advertising useful, relevant, and helpful?

I have a new post at blog@CACM titled, "Is advertising inherently deceptive?"

It discusses some of the moral and ethical qualms I have when working on personalized advertising. It attempts to start a discussion around the question of whether personalized advertising will be used for good.

An excerpt:

Let's say we build more personalization techniques and tools that allow advertisers and publishers to understand people's interests and individually target ads. How will our tools be used? Will they be used to provide better information to people about useful products and services? Or will they be used for deeper and trickier forms of deception?

Is advertising an industry fundamentally fueled by deception? Or is advertising better understood as a stream of information that, if well directed, can help people?

If you have thoughts on this topic, please contribute to the discussion, either here or over on the full post at blog@CACM.

Update: About one month later, in the October 12 issue of the New Yorker, Ken Auletta has an article, "Searching for Trouble", that describes a 2003 conflict between the COO of Viacom and the founders of Google on exactly this issue, deception in advertising. An excerpt:

[You want] salesmanship, emotion, and mystery. [Viacom COO Karmazin said], "You don't want to have people know what works. When you know what works or not, you tend to charge less money than when you have this aura and you're selling this mystique."

The Google executives thought Karmazin's method manipulated emotions and cheated advertisers.

Microsoft and Yahoo, kissing behemoths

It looks like we have a Microsoft-Yahoo deal. If you smack two amorous giants together enough times, I guess you are going to get a love child.

I doubt Google has much to fear from the laggard that likely will result. Just as two wrongs don't make a right, combining two struggling organizations is unlikely to fix dysfunction.

I hope I am wrong. If the deal provides focus rather than distractions, if it allows both organizations to rapidly iterate on, develop, and deploy products people actually want, it has some chance of succeeding.

But, as separate groups, both organizations barely can control the internal squabbling of hordes of product managers, "none f-ing getting anything done", as Carol Bartz colorfully put it. Combined, someone will have to keep these beasts from pulling on and tripping over each other while they desperately pursue the leader of the pack.

Update: Excellent commentary on the deal by Danny Sullivan, Jason Calacanis, and Saul Hansell.

Facebook versus Google?

Greg Sterling has some good thoughts on why Facebook and Google are not in competition:

Currently the use cases for Facebook and for search are quite different.

Facebook is entertaining, Facebook is fun, Facebook kills time, Facebook enables me to keep in touch with people. But Facebook, generally speaking, is not "useful" in the sense that Google is.

For its part, Google delivers information efficiently but is generally not "entertaining" or "fun."

It's very likely that the two sites will simply co-exist fulfilling different types of needs and interests ... Neither can be expected to fundamentally undermine the core business of the other.

But what are Facebook and Google's core businesses?

It is true that the uses of Facebook and Google differ. People mostly seem to go to Facebook because they find it entertaining. People mostly go to Google because it is useful.

But, the core business of both, where they get their revenue, is from advertising. And, while Google's search advertising does quite well, they have struggled much more in non-search advertising. And non-search advertising is the problem Facebok needs to solve.

Toward the end of the Fred Vogelstein's Wired article (which Greg Sterling references), Fred pinpoints the critical area of conflict:

Facebook [is] confronted with a difficult challenge: turning [their] massive user base into a sustainable business.

[Google] inked a disastrous $900 million partnership with MySpace in 2006, a failure that taught them how hard it is to make money from social networking. And privately, [Googlers] don't think Facebook's staff has the brainpower to succeed where they have failed.

"If [Facebook] found a way to monetize all of a sudden, sure, that would be a problem," says one highly placed Google executive. "But they're not going to."

Google's thin client distraction

Recently, Chris O'Brien at the San Jose Mercury News wrote:

It's getting harder every day to articulate what Google is. Is it a Web company? A software company? Something else entirely?

It's not just that it's hard to see how [Google's operating systems] fit into Google's stated mission. It's also that it's hard to explain to someone exactly what they are, or why they might, or might not, want to use them. Or to communicate why they are different from or better than any other things out there.

These new products have the whiff of engineers building things for other engineers, rather than you and me.

Even worse, these new products have the whiff of executives being unable to let go of their past battles.

For decades, Google CEO Eric Schmidt led Sun and Novell in mostly failed attempts to build thin client computers. At Google, Eric appears to be doing it again.

But Google is not a computer company. It is an advertising company. Google makes its money from advertising.

It is not as if there isn't enough to do in advertising. Despite Google's success in making search advertising more useful and helpful, most other advertising remains awful.

Fixing advertising not only would be lucrative, but also it directly fits into Google's mission to "organize the world's information and make it universally accessible and useful." At their best, ads provide useful information about interesting products and services. Right now, most contextual and display advertisements are more annoying than useful. It doesn't have to be that way.

If Google could be the solution to annoying advertising, it could reap all the rewards. Instead, Google is being led off by its generals to fight the last war.

Time effects in recommendations

The best paper award at the recent KDD 2009 conference went to Yehuda Koren's "Collaborative Filtering with Temporal Dynamics" (PDF).

The paper is a great read, not only because Yehuda is part of the team currently winning the Netflix Prize, but also because it has some surprising conclusions about how to deal with changing preferences and interests over time.

In particular, it is common in recommender systems to favor recent activity, such as more recent ratings by a user, either by only using the last N data points or by weighting more recent data more heavily. But Yehuda found that ineffective on the Netflix data:

The consistent finding was that prediction quality improves as we moderate ... time decay, reaching [the] best quality when there is no delay at all. This is despite the fact that users do change their taste and rating scale over the years.

Underweighting past action loses too much signal along with the lost noise, which is detrimental given the scarcity of data per user .... We require an accurate modeling of each point in the past, which will allow us to distinguish between persistent signal that should be captured and noise that should be isolated .... for understanding the customer ... [and] modeling other customers.

As in some of Yehuda's past work, he combines two models, one a latent factor model, the other an item-item approach. The models yielded "the best results published so far" on the Netflix data set by allowing them to represent temporal effects such as finding stronger relationships between items related in a short timeframe, handling that people tend to give higher ratings to older movies (if they bother to rate them at all), allowing for people to shift to giving higher or lower ratings on average over time, and capturing that people tend to use the same rating for multiple items rated in a short timeframe.

The paper is full of other cute tidbits too, like that they tried to detect day of the week effects -- do people rate lower on Mondays? -- but could not. They also discovered an unusual jump in the average rating in the data in 2004, which they hypothesize was due to features launched on the Netflix.com site that started showing people more movies they liked. Definitely worth a read.

Ad fatigue and relevance

At the recent Ad Auctions Workshop, I had a paper (PDF) and talk (PDF) that argued for discounting relevant advertisements more than we currently do.

To briefly summarize, if seeing bad ads causes people to look at ads less in the future (aka ad fatigue), then we should change our pricing in advertising auctions to promote relevant, useful ads. Likewise, we should charge bad ads more to compensate for the damage they cause.

The paper is not trying to be definitive. The paper only shows that ad fatigue could matter, not that it does actually matter. More work needs to be done to measure how much ad fatigue actually exists.

But, I hope this paper might motivate others to look more at ad fatigue, think more about long-term revenue instead of short-term revenue, and consider how it might be beneficial in the long-term to lower our pricing on relevant and useful advertisements.

The paper was done with Chris Meek from Microsoft Research and Max Chickering at Microsoft. It reports on a side project I did a year ago while at Microsoft Live Labs.

New Google study on speed in search results

Googler Jake Brutlag recently published a short study, "Speed Matters for Google Web Search" (PDF), which looked at how important it is to deliver and render search result pages quickly.

Specifically, Jake added very small delays (100-400ms) to the time to serve and render Google search results. He observed that even these tiny delays, which are low enough to be difficult for users to perceive, resulted in measurable drops in searches per user (declines of -0.2% to -0.6%).

Please see also my Nov 2006 post, "Marissa Mayer at Web 2.0", which summarizes a claim by Googler Marissa Mayer that Google saw a 20% drop in revenue from an accidentally introduced 500ms delay.

Update: To add to the Marissa Mayer report above, Drupal's Dries Buytaert summarized the results of a few A/B tests at Amazon, Google, and Yahoo on the impact of speed on user satisfaction. As Dries says, "Long story short: even the smallest delay kills user satisfaction."

Update: In the comments, people are asking why the effect in this study oddly appears to be an order of magnitude lower than the effects seen in previous tests. Good question there.

Update: By the way, this study is part of a broader suite of tools and tutorials Google has gathered as part of an effort to "make the web faster".

The $1M Netflix Prize has been won

An ensemble of methods from four teams has passed the criteria to win the Netflix Prize.

Other teams have 30 days to beat it, but, no matter what happens, the $1M prize will be claimed in the next month.

Congratulations to the winning team and all the competitors. It was a goal that some thought impossible without additional data, but remarkable persistence has proven the impossible possible.

Please see also my earlier post, "On the front lines of the Netflix Prize", which summarizes an article that describes some of the algorithms that brought the winning team to where it is now.

Update: In an extremely close finish, a different team, The Ensemble, took the prize. Remarkable not only to see another team qualify for the grand prize, but then to see BellKor beaten with only four minutes left in the contest.

Congratulations to all, especially those who shared knowledge and joined together to help the winning teams. The winners discovered a solution to a problem that many thought might never be solved.

Update: It appears it is still unclear who ultimately will be declared the winner of the $1M prize.

Update Good article on the current state of the contest in the New York Times.

Update: Adding to this, there is a colorful story about the nailbiting finish in a blog post from the Pragmatic Theory team (part of the "BellKor's Pragmatic Chaos" submission).

Update: Three months later, Netflix awards the prize to BellKor and starts a second contest.

Is mobile search going to be different?

In an amusingly titled WWW 2009 paper, "Computers and iPhones and Mobile Phones, oh my!" (PDF), a quartet of Googlers offer some thoughts on where mobile search may be going.

In particular, based on log analysis of iPhone searches, they claim search on mobile devices is not likely to differ from normal web search once people upgrade to the latest phones. They go on to predict that an important future feature for mobile search will be providing history and personalization synchronized across all of a person's computers and mobile devices.

Some excerpts:

We have consistently found that search patterns on an iPhone closely mimic search patterns on computers, but that mobile search behavior [on older phones] is distinctly different.

We hypothesize that this is due to the easier text entry and more advanced browser capabilities on an iPhone than on mobile phones. Thus we predict that as mobile devices become more advanced, users will treat mobile search as an extension of computer-based search, rather than approaching mobile search as a tool for a distinct subset of information needs.

For [newer] high end phones, we suggest search be a highly integrated experience with computer-based search interfaces .... in terms of personalization and available feature set .... For example, content that was searched for on a computer should be easily accessible through mobile search (through bookmarks, search summaries), and vice versa.

This similarity in queries [also] indicates that we can use the vast wealth of knowledge amassed about conventional computer based search patterns, and apply it to the emerging high-end phone search market, to quickly gain improvements in search quality and user experience.

Optimizing broad match in web advertising

A paper out of Microsoft Research, "A Data Structure for Sponsored Search" (PDF), has a couple simple but effective optimizations in it that are fun and worth thinking about.

First, a bit of explanation. When trying to match advertisers to search queries, search engines often (90%+ of the time) use broad match, which only requires a subset of the query terms to match. For example, if an advertiser bids for their ads to show on searches for "used books", the ad might also show on searches for [cheap used books], [gently used books], and [used and slightly traumatized books].

You can do this match using a normal inverted index, but it is expensive. For example, on the search for [cheap used books], you need to find all ads that want any of the terms (cheap OR used OR books), then filter out ads that wanted other terms (e.g. an ad that bid on "new books" need to be filtered out at this step). For very popular keywords such as "cheap", many ads can end up being retrieved in the first step only to be filtered in the second.

The clever and nicely simple idea in this paper is to use past data on the frequency of keywords to drop popular terms from the index whenever we possibly can. For example, let's say an advertiser bids for "the great wall of china", we can index the ad only under the lowest frequency word (e.g. "china") and not index under any of the other words. Then, on a search for [the awesomeness], we no longer have to retrieve then filter that ad.

The authors then extend this to deal with bids where all the keywords are popular (e.g. "the thing"). In the paper, they discuss a model that attempts to estimate the cost of throwing multiple words into the index (e.g. indexing "the thing") versus doing the query for each word separately.

The end result is an order of magnitude improvement in performance. All the irrelevant data you get from indexing everything wastefully pulls in many ad candidates that need to be filtered. In this case as well as others, it very much pays off to be careful about what you index.

Please see also my earlier post, "Caching, index pruning, and the query stream", that discusses a SIGIR 2008 paper out of Yahoo Research that explores some vaguely related ideas on index pruning for core search.

How much can you do with one server?

At a time when many of us are working with thousands of machines, Paul Tyma provides a remarkable example of how much you can do with just one.

Paul runs the clever Mailinator and Talkinator services. Mailinator lets people receive e-mail to arbitrary addresses under mailinator.com, mostly for disposable e-mail for avoiding spammers or annoying registration requirements. Talkinator is an instant messaging system for easily setting up and joining chat rooms.

Both are examples of removing the login friction normally associated with an application (in this case, mail and instant messaging) to discover a new application with different properties and uses.

In an older post, "The Architecture of Mailinator", Paul describes how he optimized the single server running the service to handle 6M e-mails/day. To summarize, the system is custom-built to the task, all unnecessary goo removed, and focuses on robustness while accepting a very small probability of message loss.

Recently, Paul updated that older post. Now they are processing 2M e-mails/hour on a single machine, 3.1T of data per month, and might have to expand to a second machine, not because the one machine cannot handle the load, but because two machines is the cheapest way of getting the extra bandwidth they need.

Paul also posted details on the architecture of Talkinator, which also is highly optimized to run well on a single server. If you take a peek at that, don't miss his amusing second post where he tested running on a 5W tiny plug-in server.

I am not advocating spending as much effort as Paul does on minimizing server costs. But, his work an inspirational counter-example to the common tendency to throw hardware at the problem. It is an enjoyable tale of building much by staying simple, both in the application features and the underlying architecture.

Approaching the limit on recommender systems?

An article at the upcoming UMAP 2009 conference, "I like it... I like it not: Evaluating User Ratings Noise in Recommender Systems" (PDF), looks at inconsistencies when people rates movies.

What is particularly interesting about the article is that the authors argue that "state-of-the-art recommendation algorithms" are nearing the lower bound on accuracy imposed by inconsistencies in ratings, which they and previous work refer to as the "magic barrier".

Natural variability in people's opinions limits how accurate recommender systems can be. According to this paper, we may be almost at that limit already.

Update: For more on this topic, it turns out one of the authors of the paper, Xavier Amatriain, has a blog post, "Netflix Prize: What if there is no Million $ ?", with a good comment thread.

On the front lines of the Netflix Prize

Robert Bell, Jim Bennett, Yehuda Koren, and Chris Volinsky have an article in the May 2008 IEEE Spectrum, "The Million Dollar Programming Prize", with fun tales of their work in the Netflix Prize along with a summary of the techniques that are performing best.

Some excerpts:

The nearest neighbor method works on the principle that a person tends to give similar ratings to similar movies. [For example, if] Joe likes three movies ... to make a prediction for him, [we] find users who also liked those movies and see what other movies they liked.

A second, complementary method scores both a given movie and viewer according to latent factors, themselves inferred from the ratings given to all the movies by all the viewers .... Factors for movies may measure comedy versus drama, action versus romance, and orientation to children versus orientation toward adults. Because the factors are determined automatically by algorithms, they may correspond to hard-to-describe concepts such as quirkiness, or they may not be interpretable by humans at all .... The model may use 20 to 40 such factors to locate each movie and viewer in a multidimensional space ... then [we predict] a viewer's rating of a movie according to that movie's score on the dimensions that person cares about most.

Neither approach is a panacea. We found that most nearest-neighbor techniques work best on 50 or fewer neighbors, which means these methods can't exploit all the information a viewer's ratings may contain. Latent-factor models have the opposite weakness: They are bad at detecting strong associations among a few closely related films, such as The Lord of the Rings trilogy.

Because these two methods are complementary, we combined them, using many versions of each in what machine-learning experts call an ensemble approach. This allowed us to build systems that were simple ... easy to code and fast to run.

Another critical innovation involved focusing on which movies a viewer rated, regardless of the scores. The idea is that someone who has rated a lot of fantasy movies will probably like the Lord of the Rings, even if that person has rated the other movies in the category somewhat low ... This approach nicely complemented our other methods.

Please see also my earlier post, "Netflix Prize at KDD 2008", which points at papers with more details than the IEEE article, including another recent paper by Yehuda Koren.

The siren song of startups

Over at the blog of the Communications of the ACM, I have a new post on "The Siren Song of Startups".

The article tries to get readers thinking more carefully about why they might and might not want to join a startup.

If you like the post, you might enjoy my other posts over at blog@CACM: "Enjoying Reading Research", "What To Do With Those Idle Cores?", and "What is a Good Recommendation Algorithm?"

Yahoo CEO Carol Bartz on personalization

Interesting tidbit on personalization in a Q&A with Yahoo CEO Carol Bartz:

Yahoo! is the place where millions of people come every day to see what is happening with the people and the things that matter most to them.

That could mean what's happening in the world -- like breaking news, sports scores, stock quotes, last night's TV highlights -- and your world -- like your email, photos, groups, fantasy leagues.

Based on what we know about you ... we can bring you both those worlds. So I think our clear strength is "relevance" -- whether that means knowing what weather to give you or serving up headlines you'll be interested in. It's all about really getting you.

So Yahoo wants to filter and recommend relevant content based on what it knows about you. Is this a new goal for Yahoo?

Two years ago, Yahoo co-founder Jerry Yang spoke about "better tailoring Yahoo's iconic Web portal to individual users, with the help of technology that predicts what they want."

Four years ago, Yahoo CEO Terry Semel said that one of the "four pillars" of Yahoo was "personalization technology to help users sort through vast choices to find what interests them" and Yahoo executive Lloyd Braun called it one of Yahoo's "secret weapons".

Five years ago, Yahoo CEO Terry Semel said, "Personalization will also play more of a role on the Yahoo home page in the coming months" and "painted a picture in which users could tailor the Yahoo home page to suit their particular interests," adding, "We want the home page to be totally personalized."

Making Yahoo content more relevant and useful to people would be fantastic. Personalization is a way to make the site more relevant and useful. But this idea clearly has been around the halls of Yahoo for some time. The key is going to be executing on it quickly.

Please see also my May 2006 post, "Yahoo home page cries out for personalization".

Update: Three weeks later, Forbes Magazine quotes Carol Bartz as admitting that Yahoo's troubles have been "an execution problem ... we are really working on (moving) from 'we can do this, we can do this' to 'we did do this.'"

She also has several other quotes in that same article that seem quite promising for the future of Yahoo, including that "none of us hate ads ... we just hate crappy ads" and that we should see Yahoo doing personalization and recommendations of news and other home page content soon. Great to hear it.

Update: Seven months later, Carol Bartz pumps personalization but admits that "we have been letting great data ... fall to the floor."

Update: One year later, Carol Bartz says, "Tomorrow's Yahoo is going to be really tailored" and "I want it to learn about me ... and cull through the massive amount of information that's out there to find exactly what I want." But, as ex-Yahoo Jeremy Zawodny comments, "The problem is that Yahoo execs have been saying this for literally years now. When will it come true?"