Tuesday, January 31, 2023

How can enshittification happen?

Cory Doctorow has a great piece in Wired, "The ‘Enshittification’ of TikTok. Or how, exactly, platforms die." It's about that we regularly see companies make their product worse and worse until it hits a tipping point, then the company loses its customers and starts dying.

Enshittification eventually causes the company to die, so isn't in the best interest of the company. It's definitely not maximizing shareholder value or long-term profits. So why does it happen?

Cory Doctorow does have a bit on the why, but could use a lot more: "An enshittification strategy only succeeds if it is pursued in measured amounts ... For enshittification-addled companies, that balance is hard to strike ... Individual product managers, executives, and activist shareholders all give preference to quick returns at the cost of sustainability, and are in a race to see who can eat their seed-corn first."

That's not very satisfying though. I mean, the company dies. Execs are screwing up. Why does that happen? What can be done about it? That's the question I think needs answering.

Understanding exactly why enshittification happens is important to finding real, viable solutions. Is it purposeful or unintentional on the part of teams and company leaders? Is it inevitable or preventable? If you get the root cause wrong, you'll get the wrong solution.

My view is that enshittification is mostly unintentional. I think it's a result of A/B testing, mistakes in setting up incentives, and teams busily optimizing for what's right in front of them instead of keeping their eye on the prize.

I don't think executives intentionally drive companies into the ground. I think most execs and teams have no idea that this path they are going down will cause such long-term harm to the company. If most really don't want to destroy the company, that leads to different solutions.

Layoffs as a social contagion

Stanford Professor Jeffrey Pfeffer wrote about the recent layoffs at tech companies, saying that it hurts the company in the long-term, but CEOs can't avoid the pressure to join in.
[CEOs] know layoffs are harmful to company well-being, let alone the well-being of employees, and don’t accomplish much, but everybody is doing layoffs and their board is asking why they aren’t doing layoffs also.

The tech industry layoffs are basically an instance of social contagion, in which companies imitate what others are doing. If you look for reasons for why companies do layoffs, the reason is that everybody else is doing it ... Not particularly evidence-based.

Layoffs often do not increase stock prices, in part because layoffs can signal that a company is having difficulty. Layoffs do not increase productivity. Layoffs do not solve what is often the underlying problem, which is often an ineffective strategy ... A bad decision.

For more on the harm, please see my old 2009 post from the last time this happened, "Layoffs and tech layoffs".

Monday, December 19, 2022

Are ad-supported business models anti-consumer?

Advertising-supported businesses are harder to align with long-term customer satisfaction than subscription businesses, but they make more money if they do.

A common view is that ad-supported websites, in their drive for more ad clicks, cannot resist exploting their customers with scammy content and more and more ads.

The problem is that eventually those websites become unusable and the business fails. Take the simple case of websites that put more and more ads on the page. Sure, ad revenue goes up for a while, but people rapidly become annoyed with all the ads and leave. The business then declines.

That's not maximizing revenue or profitability. That's a business failure by execs that should have known better.

It's very tempting to use short-term metrics like ad clicks and engagement for advertising-supported businesses, which encourages doing things like increasing ad load or clickbait content. But in the long run, that hurts retention, growth, and ad revenue.

In a subscription-supported business, it's easier to get the metrics right because the goal is keeping customers subscribing. In an ad-supported business, it isn't as obvious that keeping customers around and clicking ads for years is the goal. But it's still the goal.

Ad-supported businesses will make more money if they aren't filled with scams or laden with ads. But it's easy for ad-supported businesses to get the incentives and metrics wrong, much more error-prone than for subscription-supported businesses where the metrics are more obvious. While it may be harder for executives to see, ad-supported business do better if they focus on long-term customer satisfaction, retention, and growth.

Monday, December 12, 2022

Focus on the Long-term

One of my favorite papers of all time is "Focus on the Long-Term: It's better for Users and Business" from Google Research. This paper found that Google makes more money in the long-term -- when carefully and properly measured -- by reducing advertising. Because of this work, they reduced advertising on mobile devices by 50%.

tl;dr: When you increase ads, short-term revenue goes up, but you're diving deeper into ad inventory and the average ad quality drops. Over time, this causes people to look at ads less, click on ads less, and reduces retention. If you measure using long experiments that capture those effects, you find that showing fewer ads makes less money in the short-term but more money in the long-term.

Because most A/B tests don't measure long-term effects properly and this is hard for most organizations to measure correctly, the broader implication is that most websites show too many ads to maximize long-term profits.

Saturday, December 10, 2022

ML and flooding the zone with crap

Wisdom of the few is often better than wisdom of the crowds.

If the crowd is shilled and fake, most of the data isn't useful for machine learning. To be useful, you have to pull out the scarce wisdom in the sea of noise.

Gary Marcus looked at this in his latest post, "AI's Jurassic Park moment". Gary talks about how ChatGPT makes it much cheaper to produce huge amounts of reasonable-sounding bullshit and post it on community sites, then he said:

For Stack Overflow, the issue is literally existential. If the website is flooded with worthless code examples, programmers will no longer go there, its database of over 30 million questions and answers will become untrustworthy, and the 14 year old website will die.
StackOverflow added:
Overall, because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking or looking for correct answers.

The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting. Because such answers are so easy to produce, a large number of people are posting a lot of answers.

There was a 2009 SIGIR paper, "The Wisdom of the Few", that cleverly pointed out that a lot of this is unnecessary. For recommender systems, trending algorithms, reviews, and rankers, only the best data is needed to produce high quality results. Once you use the independent, reliable, high quality opinions, adding more big data can easily make things worse. Less is more, especially in the presence of adversarial attacks on your recommender system.

When using behavior data, ask what would happen if you could sort by usefulness to the ML algorithm and users. You'd go down the sorted list, then stop at some point when the output no longer improved. That stopping point would be very early if a lot of the data is crap.

In today's world, with fake crowds and shills everywhere, wisdom of the crowds fails. Data of unknown quality or provable spam should be freely ignored. Only use reliable, independent behavior data as input to ML.

Thursday, November 24, 2022

Alternatives to Twitter

About five years ago, I moved most of my blogging from here to microblogging on Twitter.

In part that was from the shut down of Google Reader. In part I was finally giving in on the trend against long form blogging. So this blog has been pretty quiet for years.

The recent decline of Twitter has me looking for and trying alternatives.

One surprise I found is that Google News and TechMeme feel worth using more often. Both are surprisingly effective alternatives to social media. I am finding they have most of the value without much of the unpleasantness, although missing the contact with close friends.

I was also surprised to find I liked LinkedIn as a substitute for Twitter. I find most of the interactions to be fairly good there, though again missing some close friends.

So far, I have more mixed feelings about using Mastodon, Facebook, Instagram, Post, or going back to blogging as alternatives. Anyone have anything else they like? Or differing experiences?

Wednesday, November 23, 2022

Quoted in the Washington Post

I'm quoted in the Washington Post today in an article titled "It’s not your imagination: Shopping on Amazon has gotten worse."

I'm talking about how Amazon used to help (but no longer) for finding and discovering what you want to buy, saying, "The store radically changes based on customer interests, showing programming titles to a software engineer and baby toys to a new mother."

The reporter, Geoffrey Fowler, goes on to say, "This is probably how most of us imagine Amazon still works. But today advertisers are driving the experience ... The Amazon we experience today is pretty much the opposite of how Amazon used to work."

The article is critical of all the ads on Amazon now, which makes the shopping experience terrible. I think it is very hard to find things on Amazon nowadays. This happened for a well-known reason. Increasing ad load -- which is the number of ads on a web page -- will usually increase short-term revenue, but it hurts retention, ad performance, and long-term revenue. As the Washington Post reporter describes, all the ads cause people to go elsewhere when they need to shop, and that has long-term costs for Amazon.

Saturday, November 19, 2022

Experimentation and metrics

Since the early days of the Web, I've been a fan of A/B testing for promoting innovation and ideas. But A/B testing is a tool. Like any tool, it can be used well or used poorly.

A/B testing observes human behavior, which is messy and complicated. Closer to behavioral economics, the metrics represent partial information and observations. From very limited data, we need to say why humans do the crazy things they do and predict what will happen next.

When used well, A/B testing helps innovation. But A/B testing should not subjugate, binding teams to do nothing unless a key metric is passed. Rather it should be used to gain partial information about expected short and long-term costs and benefits.

For misinformation, disinformation, scams, and the impact of advertising, A/B tests get some data on short-term effects, but little on long-term benefits. Ultimately there will be an investment decision about whether to pay the expected short-term costs for the hoped for long-term benefits.

A/B testing is a powerful tool for bottom-up innovation. But it is only a tool and can be used badly. A/B data should be used to inform debate, not halt debate. And I think it should always be helping to find a way to say yes to new ideas.

Sunday, July 03, 2022

Making it more difficult to shill recommender systems

Lately I've been thinking about recommender algorithms and how they go wrong. I keep hitting examples of people arguing that we should ban the fewest accounts possible when thinking about what accounts are used by recommender systems. Why? Or why not the opposite? What's wrong with using the fewest accounts you can without degrading the perceived quality of the recommendations?

The reason this matters is that recommender systems these days are struggling with shilling. Companies are playing whack-a-mole with bad actors who just create new accounts or find new shills every time they're whacked because it's so profitable -- like free advertising -- to create fake crowds that manipulate the algorithms. Propagandists and scammers are loving it and winning. It's easy and lucrative for them.

So what's wrong with taking the opposite strategy, only using the most reliable accounts? As a thought experiment, let's say you rank order accounts by your confidence they are human, independent, not shilling, and trustworthy. Then go down the list of accounts, using their behavior data until the recommendations stop improving at a noticeable level (being careful about cold start and the long tail). Then stop. Don't use the rest. Why not do that? It'd vastly increase costs for adversaries. And it wouldn't change the perceived quality of recommendations because you've made sure it wouldn't.

The traditional approach to this is to classify accounts as spam or shills separate from how the data will be used. The classifiers minimize the error rates (false positive and false negative), then treat all borderline cases as not spam. The idea here is to do almost the opposite of that traditional approach, classify accounts as trustworthy, then use only those, ignoring anything unknown or borderline as well as known spammers and shills.

This works because how the data will be used as well as the bad incentives for spammers and propagandists are really sensitive to false negatives (letting in any manipulation at all of the recommender algorithms) but not very sensitive to false positives (accidentally not using some of the data that might have actually been fine to use). Letting in even one shill can badly impact recommenders, especially when shills target getting new content trending, but using less data of the lower quality data doesn't usually change the recommendations in ways that matter for people.

This isn't my idea or a new idea, by the way. It's actually a quite old idea, talked about in papers like TrustRank, Anti-TrustRank, and Wisdom of the Few, and similar techniques are applied already by companies like Google for dealing with web spam.

The world has changed in the last decade. Especially on social media, there is rampant manipulation of wisdom of the crowd data such as likes and shares. A big part of the problem is the algorithms like trending, search, and recommender systems that pick up the manipulated data and use it to amplify shilled content. That makes it quite profitable and easy for disinformation and scams.

Places like Amazon, Facebook, and Twitter are swimming in data, but their problem is that a lot of it is untrustworthy and shilled. But you don't need to use all the data. Toss big data happily, anything suspicious at all, false positives galore accidentally marking new accounts or borderline accounts as shills when deciding what to input to the recommender algorithms. Who cares if you do?

Saturday, March 20, 2021

Wisdom of the trusted

Flood-the-zone disinformation is a problem for crowdsourced data. Wisdom of the crowds, mass amateurization, and rejection of gatekeepers no longer works with coordinated disinformation campaigns overwhelming rankers, recommenders, and content with shills and spam.

Two decades ago, a lot of us underestimated the negative effects of lower costs for communication and information sharing. While good, it also made propaganda, shilling, and manipulation far easier, and our defenses against disinformation campaigns proved weak.

We in tech were overly idealistic about what would happen as the cost of information and communication dropped. Many thought propaganda would be harder as people could now easily access the truth.

But you can't source reviews from your customers anymore if the vast majority of reviews are paid shills. You can't rank using usage data if ratings and clicks are mostly fake.

Crowdsourced information, including web crawls, reviews, and commentary, only works when almost everyone is independent and unbiased. Coordinated disinformation breaks crowdsourcing.

Flood-the-zone shouldn't have been a surprise, but it was. Propaganda and manipulation are winning because we still treat inauthentic behavior as real.

While there is plenty of mostly-deserved love for big data, often less is more when you live in an adversarial, flood-the-zone world. Wisdom of the crowds has an assumption of independence between agents, which now has been broken by coordinated disinformation campaigns.

If you are looking at garbage, there is no information. Adding disinformation to good data purely makes things worse. It's like making a milkshake, then eying a huge putrid sack of night soil nearby. Sure, you could add some of that to what you made, but even a little is going to make it worse. If there is crap everywhere, you might want to stick with what you can prove to be good.

Polling, one of the oldest forms of crowdsourced information, has been impacted too. The trend in recent years is that low response rates and shilling make it so expensive to poll that Pew Research gets better data cheaper by forming and managing a large paid panel of trusted experts.

For those working in machine learning, for those trying to work with big data, reputation and reputable sources have to be the response in a flood-the-zone world. When most of the data is bad, how you filter your data becomes the most important thing.

We have a big challenge ahead, countering disinformation using reputation and lack of reputation. In a flood-the-zone world, most data out there is now bad to useless. Isolating the useful requires skepticism toward data, like TrustRank, starting untrusted, bad until proven good.

Reviews should discard anything even resembling a shill, giving visibility only to reviews from independent and trustworthy customers. Recommender systems and rankers should focus on the data from and related to proven sources, and weight anything unknown as questionable at best and likely worthless. Most crowdsourced data for machine learning, from clicks to content, is going to have to be viewed with skepticism.

Inauthentic behavior and coordinated disinformation campaigns have shilled wisdom of the crowd to death. For reliable big data in a flood-the-zone world, it will have to be wisdom of the trusted.

Tuesday, December 15, 2020

When will virtual reality take off? The $100 bet.

About four years ago, Professor Daniel Lemire and I made a $100 bet on how quickly virtual reality would reach a broad, mainstream market. Specifically, my side of the bet was, "Virtual reality hardware (not counting cardboard) will not sell more than 10M units/year worldwide before March 2019." He bet that it would.

In early 2020, we decided to wait settle the bet because it looked like there was some chance VR would reach 10M units/year in 2020. Because of COVID and people looking for entertainment at home, Valve's release of Half Life Alyx, Supernatural (the VR exercise program), and big pushes on consumer VR by several companies, we wanted to wait and see if it was off by just one year, if 2020 was the year.

At this point, the results are in, and it is clear VR has not reached far beyond early adopters and enthusiasts. Estimates of total hardware sales vary depending on what is considered VR hardware, but most estimates I've seen have worldwide unit sales at around 5-6M in 2020.

Barron's has a nice summary: "We’ve been talking about virtual reality for decades, but it’s gone pretty much nowhere. Despite all of our advances in tech, VR hasn’t been able to bridge the physical and digital realms in any substantial way." TechCrunch adds, "There are signs of growth though it’s clear [VR] is still a niche product."

So what went wrong? Looking back at VR hype in 2016, there were a lot of reasons to be optimistic: HoloLens from Microsoft, Sony entering VR with Playstation VR, Valve pushing hard on VR in the Steam store and with their own products, Xbox looking like it might do VR, Google showing interest in VR, and, though it always seemed like vaporware to me, there was a lot of excitement around the promises made by heavily-funded MagicLeap. It looked likely that someone would make a must-have game or other compelling use of VR that might attract tens of millions of people.

Speculating a bit, I think the issue here goes beyond just needing more time, so beyond waiting for gradual acceptance of VR and growth. I think the problem is that the non-virtual-reality experience is close enough for most purposes, making VR uncompelling to set up and use.

For example, take the virtual tourism experience of visiting the International Space Station in Google Earth. It's fun and compelling enough without virtual reality, so VR in virtual tourism only a little bit of wow to the experience. Half Life Alyx seems to me to suffer from the same problem, a fun game with some compelling content, so great to try, but not a must-have. Exercise programs like Supernatural or Beat Saber fall in the same category, fun, cool to try, but not something without okay substitutes or alternatives.

At the time we made the bet back in 2016, I said something similar about why I might lose the bet: "There are several wild cards here. For example, it is possible that much cheaper units can be made to work. It's possible that someone discovers very carefully chosen environments and software tricks fool the brain into fully accepting the virtual reality, especially for gaming, increasing the appeal and making it a must-have experience for a lot of people. As unsavory as it is, pornography is often a wild card with new technology, potentially driving adoption in ways that can determine winners and losers. A breakthrough in display (such as retinal displays) might allow virtual reality hardware that is much cheaper and lighter. Business use is another unknown where virtual reality could provide a large cost savings over physical presence. I do think there are many ways in which I could lose this bet."

Unfortunately, I don't think such must-have, compelling VR experiences exist. Perhaps at some point it will. Chris Pruett, who runs part of Oculus, speculated about that, saying: "My guess would be something that is highly immersive, that involves active motion of your body, and ... it's probably going to be something that you either play with other people or is shareable with other people." That sounds plausible to me, though, more broadly, I think it has to be a must-have experience without okay substitutes in non-VR, which is a high bar. My prediction now in 2020 would be that VR will continue to struggle for years to break out beyond enthusiasts and early adopters, at least until it has a truly must-have experience.

I think Daniel Lemire took the harder side of this bet, so I'll match his $100 donation to Wikipedia to settle the bet. Back in 2016, I did add a couple ways of making my side of the bet even harder, saying I doubted even over three years in 2016-2019 that VR would sell a total of more than 10M/units, which appears to be close, and that Google Cardboard-like devices wouldn't go beyond being just a toy, so not regularly used by tens of millions, which looks like it was correct.

And I want to thank Daniel for making this bet. Whether you are betting with the hype or against it, along with conventional wisdom or against the flow, it's hard to publicly take a stand and one way or another and be willing to be wrong, especially when big company money is betting against you. This was an interesting bet.

If you enjoyed this, you might also be interested in our 2012 bet about whether tablets will replace PCs.

Update: Daniel Lemire has a post up on his thoughts on the bet, "Virtual reality… millions but not tens of millions… yet".

Friday, December 04, 2020

Facebook and investing in the long-term

Kevin Roose, Mike Isaac and Sheera Frenkel at the New York Times had a great piece ([1] [2]) on the internal debate inside Facebook on removing disinformation:
Facebook engineers and data scientists posted the results of a series of experiments called "P(Bad for the World)." ... The team trained a machine-learning algorithm to predict posts that users would consider "bad for the world" and demote them in news feeds. In early tests, the new algorithm successfully reduced the visibility of objectionable content.

But it also lowered the number of times users opened Facebook, an internal metric known as "sessions" that executives monitor closely.

Another product, an algorithm to classify and demote "hate bait" — posts that don’t strictly violate Facebook’s hate speech rules, but that provoke a flood of hateful comments ... [Another] called "correct the record," would have retroactively notified users that they had interacted with false news and directed them to an independent fact-check ... [Both were] vetoed by policy executives who feared it would disproportionately show notifications to people who shared false news from right-wing websites.

Many rank-and-file workers and some executives ... want to do more to limit misinformation and polarizing content. [Others] fear those measures could hurt Facebook’s growth, or provoke a political backlash ... Some disillusioned employees have quit, saying they could no longer stomach working for a company whose products they considered harmful.
The article is an insightful look at the struggle inside Facebook on recommender systems for news, metrics, and short vs. long-term metrics and growth. Key is fear of harming short-term metrics like sessions per user and engagement.

Any attempt to increase quality of news or ads is going to result in a short-term reduction in metrics engagement, usage, and revenue. That's obvious and not the question to ask. The question to ask is, does it pay off in the long-term?

It's unsurprising that once you've kicked off all users who hate what Facebook has become and addicted the rest to clickbait, the remainder will use Facebook less in the short-term if you improve the quality of content.

This is just like any other investment. If you invest in any large expense, you expect your short-term profits to drop, but you're betting that your long-term profits will rise. In this case, increased news quality is an investment in bringing back lapsed users.

Even measured over weeks, sessions per user is going to take a hit with a change to news quality because users who like higher quality news already disengaged and abandoned and current heavy users won't like the change. It will take a long time to pay off.

For Facebook, reducing disinformation probably would also be an investment in other areas. Facebook is polluting society with disinformation, externalizing costs; cutting disinformation is an investment in reducing regulation risk from governments. And Facebook wants good people, and many good people are leaving ([1]) or won't even consider working there because of their practices, a considerable long-term cost on the company; cutting disinformation is an investment in recruiting and retention. So Facebook probably would see benefits beyond lapsed users.

Facebook and others need to think of reducing disinformation as an investment in the future. Eliminating scams, low quality ads, clickbait, and disinformation often will reduce short-term metrics, but is a long-term investment in quality to reduce abandons, bring back lapsed users, and in other long-term business goals. These investments take a long-time to pay off, but that's why you make investments, for the long-term payoff.

Tuesday, November 26, 2019

Papers and posting

If you haven't seen it, Adrian Colyer's excellent blog has great reviews and summaries of recent papers. Back when this blog started in 2004, there weren't many people summarizing research papers. Many more are now, which is part of why I post less now. Adrian's blog is excellent and similar to what I used to do, but I think better in many ways. You can also follow Adrian Colyer on Twitter.

While I'm talking about summarizing papers, I want to highlight two lines of work that had an impact on me in the last few years and that I think deserve much more attention. Both argue we, as in all of us in tech, are doing something important wrong.

The first argues that our metrics usually are off, specifically way too focused on short-term goals like immediate revenue. This is the work started by the fantastic Focus on the Long-term out of Google and continuing from there (including [1] [2]). Because much of what we all do is an optimization process -- ML, deep learning, recommendations, A/B testing, search, and advertising -- having the targets wrong means we are optimizing for the wrong thing.

Optimizing for the wrong thing is ubiquitous in our industry. It may, for example, cause almost everyone to show too many ads and too many low quality ads. If everyone has their metrics subtly wrong, everything we make, and especially everything in the ML community, may be aiming for the wrong target.

The second is Kate Starbird's work on disinformation campaigns. Across many recent papers, Kate argues that the traditional classifier approach to spam, trolls, and shills has been failing. Adversaries can create many accounts and enlist real humans in their disinformation effort. Knocking a few accounts away does nothing; it is like shooting bullets into a wave. Instead, it is important to look at the goals of disinformation campaigns and make them more expensive to achieve. Because shills impact so many things we do -- training data for ML and deep learning, social media, reviews, recommendations, A/B testing, search, advertising -- our failure to deal with shills means the assumptions all of these systems have about the data all being equally good are wrong, and the quality of all these systems is reduced.

Solutions are hard. I'm afraid Kate's advice on solutions is limited. But I would say solutions include whitelisting (using only experts, verified real people, or accounts that are expensive to create), recognizing likely disinformation as it starts to propagate and slowing it, and countering likely disinformation with accurate information where it appears. Those replace outdated whack-a-mole account classifiers and work across multiple accounts to counter modern disinformation campaigns. Manipulation and shilling from sophisticated adversaries is ubiquitous in our industry. Until we fix this, many of our systems produce lower quality results.

Finally, I am posting a lot less here now, so let me point to other resources for anyone who liked this blog. I still post frequently on Twitter; you can follow me there. On AI/ML, there's a lot of great writing by a lot of people, far too many to list, but I can at least list my favorites, which are Fran├žois Chollet and Ben Hamner on Twitter. On economics and econometrics, which I enjoy for adding breadth to AI/ML, my favorites are economists Noah Smith and Dina Pomeranz on Twitter.

Wednesday, May 08, 2019

Tech and tech idealism

It's been almost 2 years since my last post! I don't know if anyone is still reading this. If you are, thank you!

Why haven't I posted more? Partly it is the broad transition to microblogging, which everyone is using more than long form. But part also is that I have negative feelings about where tech has been going.

I'm a tech idealist. I think tech can and should be a force for good in the world. I have spent most of my life trying to build systems where computers are helping humans. Sometimes this is by computers sifting information that is hard for people to find on their own. Sometimes this is by computers surfacing other people that can help.

Lately, some tech companies have been favoring exploitation and deception. Data is being used to manipulate. Tech is becoming customer hostile.

I've been lucky. I have gotten to work on some amazing things. There is a joy to helping someone discover a new book they will love, a bit of knowledge added to a life. Many people feel overwhelmed by the news and information in their lives, and sorting through to find what is truly important is too hard. Ads shouldn't be so annoying and irrelevant, and, you know what, they don't have to be. I've enjoyed helping people find and discover whatever they need online.

But looking at where we are in tech now, it feels like a dot com bubble again. Get rich quick. It's not building something that people love, but get the buck. Greed feeds short-term thinking. Grab that next bonus and get out before the wreckage hits.

Tech idealism is still out there. There still are many people building things that help people. There is research, the creation of knowledge and new ways to help even more people. There are many people using computers and data for good.

And there are many new people getting into computer science, which is fantastic. Computers are a force multiplier. Computers make people more productive and more powerful. Computer science and data science are just starting to have an impact in other fields.

The interdisciplinary opportunities are everywhere and exciting. We know almost nothing about our own oceans; there are huge opportunities for discoveries in biology from undersea probes and drones. We are just starting to image the entire night sky frequently, and sifting through that data with massive computing power will forever change astronomy. The field of economics is shifting to data and behavior over theory. Archeology can be fueled by processing massive amounts of satellite imagery. In field after field, computers and data are making the once impossible possible.

Tech idealism is coming back. Something may have to come to flush away some of those just seeking quick profits. Some of the worst abuses may have to be obvious failures before they are rained in. But it will change.

Computers and data are a force multiplier, allowing people to do more than they could before. Working at massive scale, computers help us understand and discover. In long-term, tech is a force for good.

Saturday, June 24, 2017

Two decades of Amazon.com recommendations

IEEE Internet Computing just celebrated its 20th anniversary.

On its 20th anniversary, the editorial board created its first ever “The Test of Time” award. I'm honored to say they gave it to our 2003 article, "Amazon.com Recommendations: Item-to-Item Collaborative Filtering", which continues to be accessed, cited, and used in industry and research many years after its original publication.

In addition, for the 20th anniversary issue of IEEE Internet Computing, we wrote a new article, “Two Decades of Recommender Systems at Amazon.com". Some excerpts:
For two decades now, Amazon.com has been building a store for every customer. Each person who comes to Amazon.com sees it differently ... It's as if you walked into a store and the shelves started rearranging themselves, with what you might want moving to the front, and what you're unlikely to be interested in shuffling further away.

Amazon.com launched item-based collaborative filtering in 1998, enabling recommendations at a previously unseen scale for millions of customers and a catalog of millions of items. Since we wrote about the algorithm in IEEE Internet Computing in 2003, it has seen widespread use across the Web, including YouTube, Netflix, and many others.

The algorithm's success has been from its simplicity, scalability, and often surprising and useful recommendations, as well as desirable properties such as updating immediately based on new information about a customer and being able to explain why it recommended something in a way that's easily understandable.

What was described in our 2003 IEEE Internet Computing article has faced many challenges and seen much development over the years ... We describe some of the updates, improvements, and adaptations for item-based collaborative filtering, and offer our view on what the future holds for collaborative filtering, recommender systems, and personalization.


What does the future hold for recommendations? ... Discovery should be like talking with a friend who knows you, knows what you like, works with you at every step, and anticipates your needs.

Recommendations and personalization live in the sea of data we all create as we move through the world, including what we find, what we discover, and what we love ... Intelligent computer algorithms leveraging collective human intelligence ... Computers helping people help other people.

The field remains wide open. An experience for every customer ... offering surprise and delight ... is a vision none have fully realized. Much opportunity remains to add intelligence and personalization to every part of every system, creating experiences that seem like a friend that knows you, what you like, and what others like, and understands what options are out there for you.

Sunday, June 11, 2017

Quick links

Some of the tech news I found interesting lately, and you might too:
  • Jeff Bezos: "Many decisions are reversible, two-way doors. Those decisions can use a light-weight process. For those, so what if you’re wrong? .... If you’re good at course correcting, being wrong may be less costly than you think, whereas being slow is going to be expensive for sure." ([1])

  • Jeff Bezos: "I would say, a lot of the value that we’re getting from machine learning is actually happening beneath the surface. It is things like improved search results. Improved product recommendations for customers. Improved forecasting for inventory management. Literally hundreds of other things beneath the surface." ([1])

  • A good summary of Mary Meeker's 2017 report. A key highlight is saturation in smartphones and internet usage. ([1])

  • New Google AI incubator: "Investment arm aimed squarely on artificial intelligence ... will operate almost like an incubator with a shared workspace for AI startups and mentorship" ([1] [2])

  • Lots of good labeled data (reliable ground truth) is the key to success with AI ([1] [2] [3] [4])

  • AI in the real world is a lot harder than ideal conditions in part because you see crazy things like robots getting attacked by humans ([1] [2])

  • "The Google [Chrome] ad-blocker will block all advertising on sites that have a certain number of 'unacceptable ads,' according to The Wall Street Journal. That includes ads that have pop-ups, auto-playing video, and 'prestitial' count-down ads that delay the display of content." ([1])

  • Nice ACM Queue article from Google SREs on availability as a combination of subservice reliability, rapid recovery, and setting expectations ([1])

  • "Designing a [software] library to reduce cognitive load is still the exception, not the rule" ([1] [2])

  • A lesson for bigger companies, investing in the long-term with your researchers, who are often working a few years ahead of what you'll need now ([1])

  • Wow: "The Melt’s blundering trajectory is instructive ... Entrepreneurs frequently embark on these missions with vast sums of money and a deep belief in technology’s power to solve all problems — which is not always a formula for success .... They were all good people, and they all wanted good things. They just didn’t know anything about running restaurants." ([1])

  • "The once-hot social network was built on the idea that people would enjoy having anonymous conversations with people close by. That’s a fantastic concept until you remember that anonymous internet person and by definition near you are scary as hell in practice." ([1])

  • Great teardown of the Juicero, includes some excellent business advice on iterative development and testing your ideas on real customers ([1] [2])

  • "When the US government discovers a vulnerability ... it can keep it secret and use it offensively ... or it can alert the software vendor and see that the vulnerability is patched, protecting the country ... Every offensive weapon is a (potential) chink in our defense." ([1])

  • On spearfishing attacks: "By a careful design and timing of a message, it should be possible to make virtually any person click" ([1] [2])

  • Schneier on forging voices: "I don't think we're ready for this. We use people's voices to authenticate them all the time, in all sorts of different ways." ([1])

  • Facebook says, "We have had to expand our security focus... to include more subtle and insidious forms of misuse, including attempts to manipulate civic discourse and deceive people" ([1] [2])

  • Remarkable and concerning that this is possible: "By accessing accelerometer and gyroscope sensors, the Web-hosted JavaScript measures subtle changes in a phone's angle, rotation, movement speed, and similar characteristics. The data, in turn, can reveal sensitive information about the phone and its user ... [including] the keystrokes being entered" ([1])

  • Nice high level description here of the difference between what Apple and Google are doing for privacy-preserving machine learning. In brief, Apple adds noise to the data to preserve privacy, but Google learns on the device then sends the updates to the machine learned models back (much like parameters servers in deep learning). The truth is they're probably both doing both, but it's still a good thing to think about. ([1])

  • Using battery backup to optimize gas power plants by being able to skip the expensive bits for gas turbines, sitting in standby because of lengthy startup times. It's easy and practical, a nice example of low hanging fruit with major impact. ([1])

  • Good data on the projected costs of energy sources ([1])

  • Good data on the newspaper industry. There's a curious spike in ad revenue from 1980-2000 that isn't matched by subscriptions. ([1])

  • Jeff Bezos is making journalism profitable: "The Post has said that it was profitable last year — and not through cost-cutting ... The Post has gone on a hiring spree. It has hired hundreds of reporters and editors and has more than tripled its technology staff ... third straight year of double-digit revenue growth ... 'You have to be great at technology. You have to be great at monetization. But one thing I think we’re proving is that if you are, great journalism can be profitable.'" ([1])

  • How Google took over the classroom, great article, but misses that the failure of iPads was a big piece of this ([1] [2])

  • Duolingo's excellent efforts to help people learn English, which can be a tool for economic or educational advancement ([1])

  • Amazon Web Services cuts prices again, remarkable ([1])

  • Almost all cloud workloads right now are not cloud optimized, so the customers mostly moved a system built for fixed hardware resources to the cloud and then run idle a lot rather than redesigning to optimize with dynamically scaling ([1])

  • Latest version of Google Earth is impressive, definitely worth trying ([1])

  • Brent Smith and I received the first ever IEEE Internet Computing Test of Time award for our 2003 paper on Amazon's recommender system. In a new article for the IEEE Internet Computing 20th Anniversary Issue, we look back at the last two decades. ([1])

  • A virtual reality game that succeeds at taking advantage of what it can do well and what it can't to create a fully immersive experience ([1])

  • Somehow, I missed that Chris Sacca is retiring. Amazing career and influence he had, and impressive to decide to go an entire new direction now. ([1])

  • In a Stack Overflow survey, what software engineers care about, it's who they work with, what they are doing, and what they learn far more than salary. In the top five items, three are about who you work with and what you learn, one is benefits, and one is commute. But the benefits are complicated -- it's not salary, stock, and bonus -- but the top items all things related to work environment and commute, vacation, and health care. ([1])

  • Great interview with the CEO of Coursera: "Humility and the ability to listen well are the big things I look for ... If you want to understand people, you need to hear them ... [Also have] ambitious goals to lift the organization up and everybody with it. Setting goals that are ambitious but also achievable is an important skill." ([1])

  • Great quote from Jeff: "At Amazon, we've had a lot of inventions that we were very excited about, and customers didn't care at all. And believe me, those inventions were not disruptive in any way. The only thing that's disruptive is customer adoption." ([1])

  • Nice line in Dan Ariely's book Payoff: "If you really want to demotivate people, shredding their work is the way to go, but ... you can get almost all the way there simply by ignoring their efforts." ([1])

  • Xkcd points out minor changes in methodology yield radical changes in data visualizations of most unusually popular activity in a location ([1])

  • Xkcd on machine learning, disturbingly close to reality ([1])

  • Xkcd on hard problems ([1])

  • Xkcd on survivorship bias ([1])

  • Xkcd on unhelpful code reviews ([1])

  • Very funny that Burger King ran an ad with "OK, Google" and it works. Once again Xkcd was hilariously prescient about this. ([1] [2])

  • SMBC comic on bayesian inference: "Given his low priors..." ([1])

  • SMBC comic: "Then it occurred to me, hey, I've got like a sample size of one here, and it's not double blind." ([1])

  • SMBC comic on behavioral economics ([1])

  • SMBC comic: "Wait, are you going to turn my life's work into a joke about butts or something?" [1])

Sunday, April 30, 2017

All Crunchzilla tutorials now open source

All the code is now available for all the Crunchzilla coding tutorials.

Code Monster, Code Maven, and Game Maven from Crunchzilla have been used by hundreds of thousands of people around the world to experiment with learning to write computer programs.

There have been many requests to make them and available in languages other than English.

By open sourcing the Crunchzilla tutorials, I hope three things might happen:

Translations: I hope others are able to take the content and translate part or all of it into languages other than English for use in more classrooms around the world.

New lessons: New tutorials might teach programming games, working through puzzles or math problems, or perhaps a more traditional computer science curriculum aligned with a particular lesson plan.

Entirely new tutorials: Some of the ideas and techniques -- including the step-by-step learn-by-doing style, live code, informative error messages, and avoiding infinite loops in students' code -- might be useful for others.
The code was designed to be all static, so you can easily create your own version just by editing the files and then putting all the files together on your own server. There is a single JSON file that contains all the lesson content.

If you use the code for anything that helps children learn, I'd love to hear about it (please e-mail me at greg@crunchzilla.com).

Sunday, April 02, 2017

Quick links

A carefully picked list of some of the tech news I enjoyed recently:
  • So, you know that prototype we showed you? Turns out AI in real world conditions is hard. ([1] [2] [3])

  • Artificial intelligence expert Yann LeCun says, "There have been, on the face of it, impressive demonstrations, [but] those are not as impressive as they look ... They don't have common sense ... One of the things we really want to do is get machines to acquire the very large number of facts that represent the constraints of the real world just by observing it through video or other channels. That’s what would allow them to acquire common sense, in the end." ([1])

  • Genetic algorithms and neural networks are back. It feels like the 1990s all over again. ([1])

  • Bringing more novices to AI now is the way to get more experts and advances later ([1])

  • Nice results from focusing on errors that matter to people, the perceived quality of the system by humans, not theoretical accuracy ([1] [2])

  • Success often comes from trying many things: "Start ... with a hazy intuition or vision ... After a lot of trial and error they get closer and closer to discovering what their idea is ... Seeking novelty instead of objectives is risky — not every interesting thread will pay off — but ... the potential payoffs are higher." ([1])

  • Research includes people able to do things no one else can, including having data or compute at the frontier beyond what anyone else has done before ([1] [2])

  • 6.3M virtual reality headsets sold in 2016, but almost all so far just the cheap toys where you slot your smartphone in to use as the screen ([1] [2])

  • "Total [tablet] sales sinking 15.6%, year on year, with sales of 174.8M units in 2016 compared to 2015's 207.2M" ([1])

  • For the first time, more people in the US using Netflix than a DVR: "54 percent of US adults reporting they have Netflix in their households compared to the 53 percent of US adults that have DVR" ([1])

  • The Economist: "Amazon’s heady valuation resembles a self-fulfilling prophecy. The company will be able to keep spending, and its spending will keep making it more powerful" ([1])

  • "What has surprised AWS as the cloud has evolved ... I don’t think in our wildest dreams we ever thought we’d have a six- to seven-year head start" ([1])

  • ... and that is true in retail for Amazon as well ([1] [2] [3])

  • "Yahoo is perhaps most famous for destroying all of its best social properties. From its hideous deformations of Flickr and neglect of Upcoming to its starvation of Delicious and torment of GeoCities users, the company excelled at buying great things and turning them into unusable parodies of themselves. Execs seemed to profoundly misunderstand why people used the sites they bought." ([1])

  • "Google will account for 78 percent of search ad revenue in 2017, while Facebook will get 39 percent of display ad revenue. Everyone else ... is fighting over the scraps." ([1])

  • Culture is created by what you publicly reward, not what you say ([1] [2] [3])

  • "The problem with bad processes is that they institutionalize inefficiency. They ensure that things will be done the wrong way, over and over and over again" ([1] [2])

  • "Burnout begins when a worker feels overwhelmed for a sustained period of time, then apathetic and ultimately numb .... Workers who used to take the lead on projects grow taciturn during meetings. Top performers start coming in late, leaving early and watch their careers stall ... Burnout is claiming victims at work, and companies aren’t ready to cope" ([1])

  • A lot of companies have merely medium data, not big data: "Hundreds of enterprises were hugely disappointed by their useless 2 to 10TB Hadoop clusters ... Their data works better in other technologies" ([1])

  • Lack of incentives leads to poor Internet of Things security ([1])

  • As Javascript ages, it repeats many of the problems of the past: "Using data from over 133K websites, we show that 37% of them include at least one library with a known vulnerability" ([1])

  • "What are some things you wish you knew when you started programming?" ([1] [2])

  • Many Xkcd comics are both funny and prescient, and this one on encryption seems particularly relevant right now ([1])

  • Xkcd comic on friends that have an Amazon Echo ([1])

  • SMBC comic on "existential sort". Don't miss the hovertext: "Also, any list can be immediately sorted by just pretty much being fine with it the way it is." ([1])

Saturday, April 01, 2017

Book review: Radical Candor

This just came out, the book Radical Candor by Kim Scott. It's a good read on managing and focused on people. I'd recommend it if you are a manager or help others manage people.

I'd summarize it by saying it takes a teaching and mentoring approach to management, very much of the school that managers primarily exist to help the people on their team. The advice is both practical and actionable, with specific advice for running 1:1s and meetings, and focused how to encourage conversations where people strive to improve themselves as well as helping others.

Some carefully selected quotes from the book:

"It seems obvious that good bosses must care personally about the people who report directly to them ... And yet ... "

"It turns out that when people trust you and believe you care about them, they are much more likely to accept and act on your praise and criticism, tell you what they really think about what you are doing well and, more importantly, not doing so well, engage in this same behavior with one another ... embrace their role on the team, and focus on getting results"

"When you're the boss, it's awkward to ask your direct reports to tell you frankly what they think of your performance, even more awkward for them than it is for you. To help, I [ask] ... 'Is there anything I could do or stop doing that would make it easier to work with me?' ... It is essential that you ... commit to sticking with the conversation until you have a genuine response. One technique is to count to six before saying anything else, forcing them to endure the silence. The goal is not to bully but to insist on a candid discussion ... Then listen with the intent to understand ... Once you've asked your question and embraced the discomfort and understood the criticism, you have to follow up by showing that you welcome it. You have to reward the candor if you want to get more of it ... Make a chance as soon as possible ... show you're trying."

"If you can absorb the blows, the members of your team are more likely to be good bosses to their employees when they have them ... The rewards of watching people you care about flourish and then help others flourish."

"The ultimate goal of Radical Candor is to achieve results collaboratively that you could never achieve individually ... A culture of guidance ... An exemplary team ... self-correcting quality whereby most problems are solved before you are even aware of them ... Don't start by bossing people. They'll just hate you. Start by listening to them."

Sunday, February 26, 2017

More quick links

Some of the tech news I found interesting lately, and you might too:
  • "In addition to making our systems more intelligent, we have to make them more intelligible too ... AI systems to augment human capabilities ... A human-centered approach is more important than ever." ([1])

  • "Understanding the brain is a fascinating problem but ... separate from the goal of AI which is solving problems ... We don’t need to duplicate humans ... We want humans and machines to partner and do something that they cannot do on their own." ([1])

  • "Machine learning and reasoning to help doctors to understand patient outcomes -- in advance of poor outcomes ... a great deal of low-hanging fruit where even today’s AI technologies are well positioned to help ... error detection, alerting, and decision support ... could save hundreds of thousands of lives per year" ([1] [2])

  • "Google's first entirely on-device ML technology ... machine intelligence ... run on your personal phone or smartwatch" ([1])

  • Accelerometers and heart rate monitors in earbuds, clever and avoids the need for a separate wearable ([1])

  • On Google's business: "Mobile search and YouTube were the main drivers of Google’s strong performance ... Google’s market share ... is above 90 percent on mobile devices" ([1] [2] [3])

  • "AI is the next platform for Facebook right now. The company is quietly approaching this initiative with the same urgency as its previous Web-to-mobile pivot." ([1])

  • "Microsoft formed a new 5,000-person engineering and research team to focus on artificial intelligence products" ([1])

  • Qi Lu leaves Microsoft for Baidu, and Jan Pedersen leaves Microsoft for Twitter. ([1] [2])

  • Not sure how well known this is: "Facebook collects information about pages [you] visit that contain Facebook sharing buttons ... And in case that wasn’t enough, Facebook also buys data about its users’ mortgages, car ownership and shopping habits from some of the biggest commercial data brokers. Facebook uses all this data to offer marketers a chance to target ads to increasingly specific groups of people. Indeed, we found Facebook offers advertisers more than 1,300 categories for ad targeting — everything from people whose property size is less than .26 acres to households with exactly seven credit cards." ([1])

  • Interesting example for the news industry: "Doubling down on traditional journalism and investing heavily in new ways to deliver it, through smartphone apps, voice-activated speakers and e-readers. The Post’s digital effort has become the envy of the industry, with as many as 80 software engineers, developers and others working alongside reporters and editors to present the news in real time." ([1])

  • "Bezos has worked to create a culture at Amazon that’s hospitable to experimentation ... developing products customers will actually want to pay for ... experiments start small and grow over time ... a small team to experiment with the idea and find out if it’s viable ... if a team succeeds in smaller challenges, it’s given more resources and a larger challenge to tackle ... prioritize launching early over everything else ... learn as quickly as possible whether an idea that sounds good on paper is actually a good idea in the real world ... getting a product into the hands of paying customers as quickly as possible and taking their feedback seriously ... avoids wasting years working on products that don’t serve the needs of real customers." ([1])

  • New direction for the cloud, just small pieces of code running somewhere (you don't care where) and data stored somewhere (you don't care where), all auto scaled ([1] [2])

  • "Many failed ideas have been resuscitated and rebranded as successful products and services, owned and managed by people other than their originators. Behind almost every popular app or website today lie numerous shadow versions that have been sloughed away by time. Yet recognition of the group nature of the enterprise would undermine a myth that legitimizes the consolidation of profit, for the most part, among a small group of people." ([1])

  • For those of us tracking virtual reality: "While Facebook does not provide sales figures for the $599 Oculus Rift headset ... analysts believe they are slow. One research firm ... estimated the company sold only about 355,000 by the end of last year." ([1] [2] [3])

  • A surprising level of detail here on what software development is like inside of Google. I agree with most of it, and highly recommend reading at least Section 2. ([1] [2])

  • Great blog post summarizing NIPS 2016. Highlights are what wins Kaggle competitions, why deep learning works, latest twiddles to deep learning and reinforcement learning, why dialogs (chat) still doesn't work, and that Baidu has products who's only value is in the data they collect (not direct revenue, just the explore part of explore/exploit, learning how to be more effective). ([1])

  • Ease of use is badly underrated: "Using TensorFlow makes me feel like I’m not smart enough to use TensorFlow; whereas using Keras makes me feel like neural networks are easier than I realized." ([1])

  • New paper by Geoff Hinton and Jeff Dean, essentially a very large ensemble of neural networks with sparsity enforced to minimize the computational cost ([1])

  • Thoughtful comments on engineering management ([1])

  • Different people we work with in tech tend to have different ideas of what it means to get things done ([1])

  • "People with different backgrounds bring new information. Simply interacting with individuals who are different forces group members to prepare better, to anticipate alternative viewpoints and to expect that reaching consensus will take effort." ([1])

  • Meetings are expensive -- a 10 person meeting for an hour costs a few thousand dollars -- and people hate meetings too. Some good reoccurring themes here are to keep meetings small, short, write a tight agenda ahead of time, stay off your laptop and phone, and try to finish early. ([1])

  • Disappointing game theory tidbit of the day, the Joy of Destruction game shows people enjoy causing harm when they can do it without consequences ([1] [2])

  • Great data visualizations from 538, not just eye candy but convey information quickly ([1])

  • "Tesla has 1.3 billion miles of car-driving data thanks to its Autopilot-equipped vehicles that are already on the road before competitors in Detroit and Silicon Valley can roll self-driving cars off the lot. It’s a massive competitive advantage." ([1])

  • Fun details on laying undersea internet cables from Amazon Web Services Distinguished Engineer James Hamilton ([1])

  • "All future wars will begin as cyberwars" ([1])

  • Impressive plans from China's space program, probes on the far side of the moon and on Mars in the next four years ([1])

  • For those interested in education, MIT's popular and excellent Scratch has published a dataset of how people learn computational thinking ([1])

  • What Code.org has achieved is very impressive: "Trained 50,000 new K-12 computer science teachers ... More than 20 million lines of code have been written by ... more than one million K-12 students ... we expect to dramatically change the demographics of AP Computer Science this year" ([1])

  • Funny article from The Onion on having too many browser tabs open ([1])

  • SMBC comic on the universe as A/B testing ([1])

  • SMBC comic on behavioral economics and anchoring ([1])

  • SMBC comic: "The wise man was put to death in the most mathematically insulting way possible" ([1])

  • Xkcd comic on what phones are, random emotional stimuli to replace boredom with anxiety ([1])

  • Xkcd comic on being an overoptimizer ([1])