A chance to play with big data

A couple fun new data sets are being made available by the search giants.

First, in a humorously titled post, "All Our N-gram are Belong to You", folks at Google Research announced that they "processed 1,011,582,453,213 words of running text and are publishing the counts for all 1,146,580,664 five-word sequences that appear at least 40 times." Very cool.

The massive Googly data set is available on six DVDs -- probably about 30G of compressed data -- but not as a download.

Second, the new AOL Research site has posted a list of APIs and data collections from AOL.

Of most interest to me is data set of "500k User Queries Sampled Over 3 Months" that apparently includes {UserID, Query, QueryTime, ClickedRank, DestinationDomainUrl} for each of 20M queries. Drool, drool!

You know, just the other day, I was watching a Google Tech Talk where a researcher was lamenting the difficulty of getting access to big data. It is exciting to see two of the giants, Google and AOL, making this kind of data available.

Update: Sadly, AOL has now taken the 500k data set offline. This is a loss to academic research community which, until now, has had no access to this kind of data.

The move seems to be a response to a bunch of inflammatory blog posts ([1] [2] [3] [4] [5]) that make outlandish claims like:

AOL has released very private data about its users without their permission ... The ability to analyze all searches by a single user will often lead people to easily determine who the user is, and what they are up to. The data includes personal names, addresses, social security numbers and everything else someone might type into a search box.

So much for the privacy of AOL's users ... This is identity theft just waiting to happen, that's what this is.

I expect it's a matter of time before a major national newspaper prints an interview with somebody identified and embarrassed in this manner.

Nevermind that no one actually has come up with an example where someone could be identified. Just the theoretical possibility is enough to create a privacy firestorm in some people's minds.

I am as concerned about privacy as any tech geek, but most of my concern is focused on things like millions of credit cards being leaked and millions of social security numbers being lost.

If someone comes up with a clear example of a privacy violation from this AOL data, I would be convinced. Until then, this looks to me like the mob of the blogosphere getting distracted in the shadows and missing the big privacy picture.

Unfortunately, the research community now will be denied a tool that could have helped push forward the state of information retrieval. Research that could have been accelerated will now be stalled. We all will suffer from the loss.

Update: I have brief quotes in articles in the New York Times and the Red Herring on this topic.

Update: This morning, in a front page article, New York Times reporters track down a specific AOL user using the released data set and ask her about her AOL searches.

Update: It now appears the entire AOL Research site has been taken offline, including access to their publications, other data sets, and APIs. That is disappointing.

Update: The NYT reports that AOL's CTO, a senior researcher, and a senior manager have been dismissed. It appears AOL Research is being shut down.

Some may cheer AOL getting a firm spanking over this privacy issue, but I think the long-term costs are grave. I suspect this pretty much eliminates any future access for the academic research community to large scale data sets. After this, the only work on big data will be at the search giants.

Hindering academic research will slow progress on building the next generation of search. It is hard to measure the cost of difficulty finding the information you need -- the productivity loss of a few minutes a day over millions of people is difficult to measure -- but it is a cost we will all be paying.

Update: Annalee Newitz at Wired Magazine adds some perceptive with a list of severe privacy violations from the past and present. [via Bruce Schneier]

Update: A couple days later, Katie Hafner at the NYT writes that "Researchers Yearn to Use AOL Logs, but They Hesitate".

Update Seven weeks later, the NYT also adds perceptive by writing about recent examples of severe privacy violations including "2.6 million current and former Circuit City credit card account holders", "names, Social Security numbers, and dates of birth on roughly 28 million veterans", "names, addresses and credit and debit card numbers of some 243,000 customers of Hotels.com", "names and Social Security numbers — and in some instances medical histories — of some 51,000 current and former patients of PSA HealthCare", ChoicePoint's leak of 145,000 accounts, and CardSystems' leak of 40M credit card numbers. Read more at the "Chronology of Data Breaches".

Update: Seven months later, Henry Blodget writes:

ISPs happily sell clickstream data ... It's a big business.

They don't sell your name--just your clicks -- but the clicks are tied to you as a specific user (User 1, User 2, etc.).

How much are your clicks worth? According to [Compete CTO] David [Cancel], about 40 cents a month per user (per customer) .. .and he estimates that there are 10-12 big buyers of this data. In other words, your ISP is probably making about $5 a month ($60 a year) off your clickstreams.

Someone [in the audience] points out that this is just as bad as the AOL search thing. It's much more! David says -- his excited eyes indicating that he's a happy customer. Someone else observes that the benefits/drawbacks of this are in the eye of the beholder: for the ISPs it's awesome.