Book excerpt: Overview from the book proposal

(This is an excerpt from the book proposal for my unpublished book, "Algorithms and Misinformation: Why Wisdom of the Crowds Failed the Internet and How to Fix It")

Without most of us even realizing it, algorithms determine what we see everyday on the internet.

Computer programs pick which videos you’ll watch next on TikTok and YouTube. When you go to Facebook and Twitter, algorithms pick which news stories you’ll read. When it’s movie night, algorithms dictate what you’ll watch on Netflix based on what you watched in the past. Everywhere you look, algorithms decide what you see.

When done well, these computer programs have enormous value, helping people find what they need quickly and easily. It’s hard to find what you are looking for with so much out there. Algorithms filter through everything, tossing bad options away with wild abandon, to bring rare gems right to you.

Imagine you’re looking for a book. When you go to Amazon and start searching, algorithms are what filter through all the world’s books for you. But not only that. Algorithms also look at what books people seem most interested in and then bring you the very best choices based on what other customers bought. By quickly filtering through millions of options, computers help people discover things they never would have been able to find on their own.

These algorithms make recommendations in much the same way that you would. Suppose you have a friend who asks you to recommend a good book for her to read. You might ask yourself, what do you know about her? Does she like fiction or nonfiction? Which authors does she like? What books did she read in the past few months? With a little information about your friend’s tastes, you might narrow things down. Perhaps she would like this well-reviewed mystery book? It has some similar themes to a book she enjoyed last year.

Algorithms combine opinions, likes, and dislikes from millions of people. The seminal book The Wisdom of Crowds popularized the idea that combining the opinions of many random people often gives useful results. What algorithms do is bring together the wisdom of crowds at massive scale. One way they do this is by distilling thousands of customer reviews so you can easily gauge the average review of a movie or video game before you sink time and money into it. Another way is by showing you that customers who bought this also bought that. When algorithms pick what you see on the internet, they use wisdom of the crowds.

Something changed a few years ago. Wisdom of the crowds failed. Algorithms that use wisdom of the crowds started causing harm. Across the internet, algorithms that choose what people see started showing more spam, misinformation, and propaganda.

What happened? In the same way a swindler on a street corner will stack the crowd with collaborators who loudly shill the supposed wonders of their offerings, wisdom of the crowd algorithms got fooled into promoting misinformation, scams, and frauds. With the simple ease of creating many online accounts, a fraudster can pretend to be an entire crowd of people online. A fake crowd gives scammers a megaphone that they can use to amplify their own voice as they drown out the voices of others.

Search and recommendation algorithms across the internet were fooled by these fake crowds. Before the 2020 election in the United States, foreign adversaries posted propaganda to social media, then pretended to be large numbers of Americans liking and resharing, fooling the algorithms into amplifying their posts. 140 million people in the United States saw this propaganda, many of whom were voters. In 2019, the largest pages on social media for Christian Americans, such as “Be Happy Enjoy Life” and “Jesus is my Lord”, were controlled by foreign operatives pretending to be Americans. These troll farms shilled recommendation, search, and trending algorithms, getting top placement for their posts and high visibility for their groups, reaching 75 million people. Scammers manipulated wisdom of the crowd algorithms with shills to promote their bogus cures during the COVID-19 global pandemic. In 2021, the US Surgeon General was so alarmed by health misinformation on the internet that he warned of increased illness and death if it continued.

Misinformation and disinformation are now the biggest problems on the internet. It is cheap and easy for scammers and propagandists to get seen by millions. Just create a few hundred accounts, have them like and share your stuff to create the illusion of popularity, and wisdom of the crowd algorithms will amplify whatever you like. Even once many companies realized the algorithms had gone wrong, many failed to fix it.

This book is about fixing misinformation on the internet by fixing the algorithms that promote misinformation. Misinformation, scams, and propaganda are ubiquitous on the internet. Algorithms including trending, recommendations, and search rankers amplify misinformation, giving it much further reach and making it far more effective.

But the reason why algorithms amplify misinformation is not what you think. As this book shows, the process of how big tech companies optimize algorithms is what causes those algorithms to promote misinformation. Diving deep inside the tech companies to understand how they build their algorithms is the key to finding practical solutions.

This book could only be written by an insider with an eye toward how the biggest tech companies operate. That’s because it’s necessary to not only understand the artificial intelligence technology behind the algorithms that pick what people see on the internet, but also understand the business incentives inside these companies when teams build and optimize these algorithms.

When I invented Amazon’s recommendation algorithm, our team was idealistic about what would happen next. We saw algorithms as a tool to help people. Find a great book. Enjoy some new music. Discover new things. No matter what you are looking for, someone out there probably already found it. Wisdom of the crowd algorithms share what people found with other people who might enjoy it. We hoped for an internet that would be a joyful playground of knowledge and discovery.

In the years since, and in my journeys through other tech companies, I have seen how algorithms can go terribly wrong. It can happen easily. It can happen unintentionally. Like taking the wrong path in a dark forest, small steps lead to bigger problems. When algorithms go wrong, we need experts like me who can see realistic ways to correct the root causes behind the problems.

Solutions to what is now the world’s algorithm problem require interdisciplinary expertise in business, technology, management, and policy. I am an artificial intelligence expert, invented Amazon’s recommendation algorithm, and have thirty-two patents on search and recommendation algorithms. I also have a Stanford MBA, worked with executives at Amazon, Microsoft, and Netflix, and am an expert on how tech companies manage, measure, and reward teams working on wisdom of the crowd algorithms. Past books have failed to offer solutions because authors have lacked the insider knowledge, and often the technical and business expertise, to solve the problems causing misinformation and disinformation. Only with a deep understanding of the technology and business will it be possible to find solutions that not only will work, but also will be embraced by business, government, and technology leaders.

This book walks readers through how these algorithms are built, what they are trying to do, and how they go wrong. I reveal what it is like day-to-day to work on these algorithms inside the biggest tech companies. For example, I describe how the algorithms are gradually optimized over time. That leads to the surprising conclusion that critical to what the algorithms show people is not the algorithms themselves but the metrics companies pick for judging if the algorithms are doing their job well. I show how easy it is for attempts to improve algorithms to instead go terribly wrong. Seemingly unrelated decisions such as how people are promoted can not only cause algorithms to amplify misinformation, but also hurt customers and the long-term profitability of the company.

Readers need to know both why the algorithms caused harm and why some companies failed to fix the problems. By looking at what major tech companies have done and failed to do, readers see the root causes of the massive spread of misinformation and disinformation on the internet. Some companies have invested in fixing their algorithms and prospered. Some companies failed to fix their algorithms and suffered higher costs as misinformation and scams grew. By comparing companies that have had more success with those that have not, readers discover how some companies keep fraudsters from manipulating their algorithms and why others fail.

Other books have described misinformation and disinformation on the internet, but no other book offers practical solutions. This book explains why algorithms promote misinformation with key insights into what makes misinformation cost effective for fraudsters. This book describes what tempts giant tech companies to allow misinformation on their platforms and how that eventually hurts the companies and their customers. Importantly, this book provides strong evidence that companies would benefit from fixing their algorithms, establishing that companies make more money when they fix their algorithms to stop scams, propaganda, and misinformation. From this book, consumers, managers, and policy makers not only will know why algorithms go wrong, but also will be equipped with solutions and ready to push for change.

This is the story of what went wrong and how it can be fixed as told by people who were there. I bring together rare expertise to shine a light on how to solve the greatest problem on the internet today. This book is a guide inside how the world’s biggest technology companies build their algorithms, why those algorithms can go wrong, and how to fix it.

Book excerpt: The irresistible lure of an unlocked house

(This is an excerpt from drafts of my unpublished book, "Algorithms and Misinformation: Why Wisdom of the Crowds Failed the Internet and How to Fix It")

Bad incentives and bad metrics create an opportunity. They are what allow bad guys to come in and take root. Scammers and propagandists can take advantage of poorly optimized algorithms to make algorithms promote whatever misinformation they like.

Adversaries outside of these companies see wisdom of the crowd algorithms as an opportunity for free advertising. By manipulating algorithms with fake crowds, such as an astroturf campaign of controlled accounts and bots pretending to be real people, bad actors can feign popularity. Wisdom of the crowds summarizes opinions of the crowd. If the crowd is full of shills, the opinions will be skewed in whatever direction the shills like.

There is a massive underground economy around purchasing five star reviews on Amazon — as well as offering one star reviews for competing products — that allows counterfeiters and fraudsters to purchase whatever reputation they like for questionable and even dangerous products. Third-party merchants selling counterfeit, fraudulent, or other illicit goods with very high profit margins buy reviews from these services, feigning high quality to unwitting Amazon customers. If they are caught, they simply create a new account, list all their items again, and buy more fake reviews.

Get-rich-quick scammers and questionable vitamin supplement dealers can buy fake crowds of bogus accounts on social media that like and share their false promises. Buying fake crowds of followers on social media that like and share your content is a mature service now with dealers offering access to thousands of accounts for a few hundred dollars. Scammers rely on these fake crowds shilling their wares to fool algorithms into promoting their scams.

Foreign operatives have buildings full of people, each employee sitting at a desk pretending to be hundreds of Americans at once. They spend long days at work on social media with their multitude of fake accounts, commenting, liking, following, and sharing, all with the goal of pushing their disinformation and propaganda. The propaganda effort was so successful that, by 2019, some of the largest pages on social media were controlled by foreign governments with interests not aligned with the United States. Using their multitude of fake accounts, they were able to fool social media algorithms into recommending their pages and posts. Hundreds of millions of Americans saw their propaganda.

It is cheap to buy fake crowds and swamp wisdom of the crowd algorithms with bogus data about what is popular. When the crowd isn’t real, the algorithms don’t work. Wisdom of the crowd relies on crowds of independent, real people. Fake crowds full of shills means there is no wisdom in that crowd.

When algorithms amplify scams and disinformation, it may increase a platform’s engagement metrics for the moment. But, in the long-run, the bad actors win and the company loses. It is easy for people inside of tech companies to unwittingly optimize their algorithms in ways that help scammers and propagandists and hurt customers.

A summary of my book

My book is the untold story of the algorithms that shape our lives, how they went terribly wrong, and how to fix them.

Most people now have at least a vague idea that algorithms choose what we see on our favorite online platforms. On Amazon they recommend millions of products. On Facebook they predict whether we’re more likely to click on a cute animal video or a rant about Donald Trump. At Netflix, Spotify, Twitter, YouTube, Instagram, TikTok, and every other site on the internet, they serve billions of users with billions of recommendations. But most people don’t know how all those algorithms really work — or why in recent years they began filling our screens with misinformation.

Other books have described the abundant misinformation, scams, and propaganda on many platforms, but this is the first to offer practical fixes to misinformation and disinformation across the entire internet by focusing on how and why algorithms amplify harmful content. This book offers solutions to what has become the biggest problem on the internet, using insider knowledge from my 30 years of experience in artificial intelligence, recommender systems, search, advertising, online experimentation, and metrics, including many years at Amazon, Microsoft, and startups.

Many assume “the problem with algorithms” is a tech problem, but it’s actually an incentives problem. Solutions must begin with the incentives driving the executives who run platforms, the investors who fund them, the engineers who build and optimize algorithms, and the content creators who do whatever it takes to maximize their own visibility. Ultimately, this is a book about people and how people optimize algorithms.

Equipped with insider knowledge of why these algorithms do what they do, readers will finish this book with renewed hope, practical solutions, and ready to push for change.

(this was a summary of my book, and I will be posting more excerpts from the book here)

Book excerpt: The problem is fake crowds

(This is an excerpt from my book. Please let me know if you like it and want more.)

It is usually unintentional. Companies don’t intend for their websites to fill with spam. Companies don’t intend for their algorithms to amplify propagandists, shills, and scammers.

It can happen just from overlooking the problem then build up over time. Bad actors come in, the problem grows and grows, and eventually becomes difficult and costly to stop.

For the bad guys, the incentives are huge. Get your post trending, and a lot of people will see it. If your product is the first thing people see when they search, you will get a lot of sales. When algorithms recommend your content to people, that means a lot more people will see you. It’s like free advertising.

Adversaries will attack algorithms. They will pay people to offer positive reviews. They will create fake crowds consisting of hundreds of fake accounts, all together liking and sharing their brilliant posts, all together saying how great they are. If wisdom of the crowd algorithms treat these fake crowds as real, the recommendations will be shilled, spammy, and scammy.

Allow the bad guys to create fake crowds and the algorithms will make terrible recommendations. Algorithms try to help people find what they need. They try to show just the right thing to customers at just the right time. But fake crowds make that impossible. Facebook suffers from this problem. An internal study at Facebook looked at why Facebook couldn’t retain young adults. Young people consistently described Facebook as “boring, misleading, and negative” and complained that “they often have to get past irrelevant content to get to what matters.”

Customers won’t stick around if what they see is mostly useless scams. Nowadays, Facebook’s business has stalled because of problems with growth and retention, especially with young adults. Twitter's audience and revenue has cratered.

Bad, manipulated, shilled data means bad recommendations. People won’t like what they are seeing, and they won’t stay around.

Kate Conger wrote in the New York Times about why tech companies sometimes underestimate how bad problems with spam, misinformation, propaganda, and scams will get if neglected. In the early years of Twitter, “they believed that any reprehensible content would be countered or drowned out by other users.” Jason Goldman, who was very early at Twitter, described “a certain amount of idealistic zeal” that they all had, a belief that the crowds would filter out bad content and regulate discussion in the town square.

It wasn’t long until adversaries took advantage of their naiveté: “In September 2016, a Russian troll farm quietly created 2,700 fake Twitter profiles” which they used to shill and promote whatever content they liked, including attempting to manipulate the upcoming US presidential election.

On Facebook, “One Russian- run Facebook page, Heart of Texas, attracted hundreds of thousands of followers by cultivating a narrow, aggrieved identity,” Max Fisher wrote in The Chaos Machine. “‘Like if you agree,’ captioned a viral map with all other states marked ‘awful’ or ‘boring,’ alongside text urging secession from the morally impure union. Some posts presented Texas identity as under siege (‘Like & share if you agree that Texas is a Christian state’).”

Twitter was born around lofty goals of the power of wisdom of the crowds to fix problems. But the founders were naive about how bad the problems could get with bad actors creating fake accounts and controlling multiple accounts. By pretending to be many people, adversaries could effectively vote many times, and give the appearance of a groundswell of faked support and popularity to anything they liked. Twitter’s algorithms would then dutifully pick up the shilled content as trending or popular and amplify it further.

Twitter later “rolled out new policies that were intended to prevent the spread of misinformation,” started taking action against at least some of the bot networks and controlled accounts, and even “banned all forms of political advertising.” That early idealism that “the tweets must flow” and that wisdom of the crowds would take care of all problems was crushed under a flood of manipulated fake accounts.

Bad actors manipulate wisdom of the crowds because it is lucrative to do so. For state actors, propaganda on social media is cheaper than ever. Creating fake crowds feigns popularity for their propaganda, confuses the truth in a flood of claims and counterclaims, and silences opposition. For scammers, wisdom of the crowds algorithms are like free advertising. Just by creating a few hundred fake accounts or by paying others to help shill, they can wrap scams or outright fraud in a veneer of faked reliability and usefulness.

“Successfully gaming the algorithm can make the difference between reaching an audience of millions – or shouting into the wind,” wrote Julia Carrie Wong in the Guardian. Successfully manipulating wisdom of the crowds data tricks trending and recommender algorithms into amplifying. Getting into trending, the top search results, or getting recommended by manipulating using fake and controlled accounts can be a lot cheaper and more effective than buying advertising.

“In addition to distorting the public’s perception of how popular a piece of content is,” Wong wrote, “fake engagement can influence how that content performs in the all-important news feed algorithm.” With fake accounts, bad actors can fake likes and shares, creating fake engagement and fake popularity, and fooling the algorithms into amplifying. “It is a kind of counterfeit currency in Facebook’s attention marketplace.”

“Fake engagement refers to things such as likes, shares, and comments that have been bought or otherwise inauthentically generated on the platform,” Karen Hao wrote in MIT Technology Review. It’s easy to do. “Fake likes and shares [are] produced by automated bots and used to drive up someone’s popularity.”

“Automation, scalability, and anonymity are hallmarks of computational propaganda,” wrote University of Oxford Professor Philip Howard in his recent book Lie Machines. “Programmers who set up vast networks” of shills and bots “have a disproportionate share of the public conversation because of the fake user accounts they control.” For example, “dozens of fake accounts all posing as engaged citizens, down- voting unsympathetic points of view and steering a conversation in the service of some ideological agenda— a key activity in what has come to be known as political astroturfing. Ordinary people who log onto these forums may believe that they are receiving a legitimate signal of public opinion on a topic when they are in effect being fed a narrative by a secret marketing campaign.” Fake crowds create a fake “impression that there is public consensus.” And by manipulating wisdom of the crowds algorithms, adversaries “control the most valuable resource possible … our attention.”

The most important part is at the beginning. Let’s say there is a new post full of misinformation. No one has seen it yet. What it needs is to look popular. What it needs is a lot of clicks, likes, and shares. If you control a few hundred accounts, all you need to do is have them all engage with your new post around the same time. And wow! Suddenly you look popular!

Real people join in later. It is true that real people share misinformation and spread it further. But the critical part is at the start. Fake crowds make something new look popular. It isn’t real. It’s not real people liking and sharing the misinformation. But it works. The algorithms see all the likes and shares. The algorithms think the post is popular. The algorithms amplify the misinformation. Once the algorithms amplify, a lot of real people see the shilled post. It is true that there is authentic engagement from real people. But most important is how everything got started, shilling using fake crowds.

When adversaries shill wisdom of the crowd algorithms, they replace the genuinely popular with whatever they like. This makes the experience worse and eventually hurts growth, retention, and corporate profits. These long-term costs are subtle enough that many tech companies often miss them until the costs become large.

Ranking algorithms use wisdom of the crowds to determine what is popular and interesting. Wisdom of the crowds requires independent opinions. You don't have independent opinions when there is coordinated shilling by adversaries, scammers, and propagandists. Faked crowds make trending, search, and recommendation algorithms useless. To be useful, the algorithms have to use what real people actually like.

Book excerpt: Mark as spam, the long fight to keep emails and texts useful

(This is an excerpt from my book. Please let me know if you like it and want more.)

The first email on the internet was sent in 1971. Back then, the internet was a small place used only by a few geeky researchers affiliated with ARPANET, an obscure project at the Department of Defence.

Oh how the internet has grown. Five billion people now use the internet, including nearly everyone in the United States, as well as most of the world. A lot of the time, we use the internet to communicate with our friends using email and text messaging.

As the internet usage grew, so did the profit motive. The first email spam was sent in 1978, an advertisement for mainframe computers. By the mid-1990s, as more and more people started using the internet, email spam became ubiquitous. Sending a spam message to millions of people could get a lot of attention and earn spammers a lot of money. All it took was a small percentage of the people responding.

It got to the point that, by the early 2000s, email was becoming difficult to use because of the time-consuming distraction of dealing with unwanted spam. The world needed solutions.

The problem is aggravated by executives often unwittingly measuring the goals of their marketing teams by whether people click on their emails, which has unintended harmful consequences. If you measure teams by how many clicks they get on their emails, the teams have a strong incentive to send as much email as possible. And that means customers get annoyed by all the emails and start marking it as spam. This long-term cost – that you might not be able to send email anymore to customers if you send them too much email – needed to be part of the goals of any team sending email to customers.

The bigger email spam problem was that spam worked for the bad guys. When bad actors can make money by sending spam emails, you get a lot of spam emails. Spammers could make a lot of money by evading spam filters. So they worked hard to trick spam filters by, for example, using misspellings to get past keyword detection.

In the early 2000s, email was dying under spam. It was bad out there. Spam filtering algorithms were in an arms race against bad actors who tried everything to get around them. Anti-spam algorithms filtered out spammy keywords, so the bad guys used misspelling and hordes of fake accounts to get back in that inbox. The good guys adapted to the latest tactics, then the bad guys found new tricks.

What finally fixed it was to make email spam unprofitable. If you never see spam, it is like it doesn't exist for you. Spammers spam because they make money. If it becomes more difficult to make money, there will be fewer spammers sending fewer scams to your inbox. But how can you make spam less profitable?

What worked was reputation. Much like TrustRank, known spammers and unknown senders of email tend to be unreliable, and reliable and well-known internet domains tend to not send spam. Reliable companies and real people should be able to send email. New accounts created on new internet domains, especially if they have sent spam before, probably should not be able to send email. Treating every email from unknown or unreliable sources with great suspicion, and skipping the inbox, means most people nowadays rarely see email and text spam, merely an occasional nuisance today.

Email spam is barely profitable these days for spammers. Reducing the payoff from spamming changes the economics of spam. To discourage bad behaviors, make them less profitable.

Cory Doctorow on enshittification

Another good piece by Cory on enshittification, with details about Facebook, some on how A/B testing optimizes for enshittification, and updated with how Musk's Twitter is impatiently racing to enshittify. An excerpt from Cory's piece:

Enshittification is the process by which a platform lures in and then captures end users (stage one), who serve as bait for business customers, who are also captured (stage two) whereupon the platform rug-pulls both groups and allocates all the value they generate and exchange to itself (stage three).

It was a long con. Platform operators and their investors have been willing to throw away billions convincing end-users and business customers to lock themselves in until it was time for the pig-butchering to begin. They financed expensive forays into additional features and complementary products meant to increase user lock-in, raising the switching costs for users who were tempted to leave.

Tech platforms are equipped with a million knobs on their back-ends, and platform operators can endlessly twiddle those knobs, altering the business logic from moment to moment, turning the system into an endlessly shifting quagmire where neither users nor business customers can ever be sure whether they're getting a fair deal.

For users, this meant that their feeds were increasingly populated with payola-boosted content from advertisers and pay-to-play publishers ... Twiddling lets Facebook fine-tune its approach. If a user starts to wean themself off Facebook, the algorithm (TM) can put more content the user has asked to see in the feed. When the user's participation returns to higher levels, Facebook can draw down the share of desirable content again, replacing it with monetizable content. This is done minutely, behind the scenes, automatically, and quickly. In any shell game, the quickness of the hand deceives the eye.

If a user starts to wean themself off Facebook, the algorithm (TM) can put more content the user has asked to see in the feed. When the user's participation returns to higher levels, Facebook can draw down the share of desirable content again, replacing it with monetizable content. This is done minutely, behind the scenes, automatically, and quickly. In any shell game, the quickness of the hand deceives the eye ... This is the final stage of enshittification: withdrawing surpluses from end-users and business customers, leaving behind the minimum homeopathic quantum of value for each needed to keep them locked to the platform, generating value that can be extracted and diverted to platform shareholders.

But this is a brittle equilibrium to maintain. The difference between "God, I hate this place but I just can't leave it" and "Holy shit, this sucks, I'm outta here" is razor-thin. All it takes is one privacy scandal, one livestreamed mass-shooting, one whistleblower dump, and people bolt for the exits. This kicks off a death-spiral: as users and business customers leave, the platform's shareholders demand that they squeeze the remaining population harder to make up for the loss.

As much as Cory talks about it here, I do think the role of A/B testing in enshittification is understated. Teams can unintentionally enshitify just with repeated A/B testing and optimizing for the metrics they are told to optimize for. It doesn't necessarily take malice, certainly not on the part of everyone at the company, just A/B testing, bad incentive systems for bonuses and promotions, and bad metrics like engagement.

To stop disinformation, stop astroturf

(this is a version of an excerpt from my book, if you like it please let me know)

There's a lot of discussion of removing disinformation as censorship lately. I think this gets the problem wrong. The problem is using many accounts that you control to act like a megaphone for your speech. Platforms can prevent disinformation by preventing bad actors from astroturfing popularity using faked crowds.

Governments regulating speech is fraught with peril. But disinformation campaigns don't work by using normal speech. They work by creating thousands of controlled accounts that like and share their own content, creating the appearance of popularity, which algorithms like search, trending, and recommendations then pick up and amplify further.

There's no right to create x1000 accounts for yourself and shout down everyone else. That's not how social media is supposed to work. And it's definitely not how wisdom of crowds is supposed to work. In wisdom of crowds, every voice has to be independent for the result to be valid. Search rankers, trending algorithms, and recommender systems are all based on wisdom of the crowds.

Regulators should focus not on specific posts or accounts, but on manipulation of the platforms by creating many accounts. It's fraudulent manipulation of platforms by spoofing what is popular. Astroturfing causes disinformation, not individuals posting what they think.

Book excerpt: The problem is not the algorithm

(This is an excerpt from the draft of my book. Please let me know if you like it and want more of these.)

“The Algorithm,” in scare quotes, is an oft-attacked target. But this obscures more than it informs.

It creates the image of some mysterious algorithm, intelligent computers controlling our lives. It makes us feel out of control. After all, if the problem is “the algorithm”, who is to blame?

When news articles talk about lies, scams, and disinformation, they often blame some all-powerful, mysterious algorithm as the source of the troubles. Scary artificial intelligence controls what we see, they say. That grants independence, agency, and power where none exists. It shifts responsibility away from the companies and teams that create and tune these algorithms and feed them the data that causes them to do what they do.

It's wrong to blame algorithms. People are responsible for the algorithms. Teams working on these algorithms and the companies that use them in their products have complete control over the algorithms. Every day, teams make choices on tuning the algorithms and what data goes into the algorithms that change what is emphasized and what is amplified. The algorithm is nothing but a tool, a tool people can control and use any way they like.

It is important to demystify algorithms. Anyone can understand what these algorithms do and why they do it. “While the phrase ‘the algorithm’ has taken on sinister, even mythical overtones, it is, at its most basic level, a system that decides a post’s position on the news feed based on predictions about each user’s preferences and tendencies,” wrote the Washington Post, in an article “How Facebook Shapes Your Feed.” How people tune and optimize the algorithms determines “what sorts of content thrive on the world’s largest social network and what types languish.”

We are in control. We are in control because “different approaches to the algorithm can dramatically alter the categories of content that tend to flourish.” Choices that teams and companies make about how to tune wisdom of the crowd algorithms make an enormous difference for what billions of people see every day.

You can think of all the choices for tuning the algorithms as a bunch of knobs you can turn. Turn that knob to make the algorithm show some stuff more and other stuff less.

When I was working at Amazon many years ago, an important knob we thought hard about turning was how much new items were recommended. When recommending books, one choice would tend to show people more older books that they might like. Another choice we could make would show people more new releases, such as new books that came out in the last year or two. On the one hand, people are particularly unlikely to know about a new release, and new books, especially by an author or in a genre you tend to read, can be particularly interesting to hear about. On the other hand, if you go by how likely you are to buy a book, maybe the algorithm should recommend older books. Help people discover something new or maximize sales today, our team had a choice in how to tune the algorithm.

Wisdom of the crowds works by summarizing people’s opinions. Another way that people control the algorithms is through the information about what people like, buy, and find interesting and useful. For example, if many people post positive reviews of a new movie, the average review of that movie might be very high. Algorithms use those positive reviews. This movie looks popular! People who haven’t seen it yet might want to hear about it. And people who watched similar other movies, such as movies in the same genre or with the same actors, might be particularly interested in hearing about this new movie.

The algorithms summarize what people are doing. They calculate and collate what people like and don’t like. What people like determines what the algorithms recommend. The data about what people like controls the algorithms.

But that means that people can change what the algorithms do through changing the data about what it seems like people like. For example, let’s say someone wants to sell more of their cheap flashlights, and they don’t really care about the ethics of how they get more sales. So they pay for hundreds of people to rate their flashlight with a 5-star review on Amazon.

If Amazon uses those shilled 5-star reviews in their recommendation engines and search rankers, those algorithms will mistakenly believe that hundreds of people think the flashlights are great. Everyone will see and buy the terrible flashlights. The bad guys win.

If Amazon chooses to treat that data as inauthentic, faked, bought-and-paid-for, and then ignores those hundreds of paid reviews, that poorly-made flashlight is far less likely to be shown to and bought by Amazon customers. After all, most real people don’t like that cheap flashlight. The bad guys tried hard to fake being popular, but they lost in the end.

The choice of what data is used and what is discarded makes an enormous difference in what is amplified and what people are likely to see. And since wisdom of the crowd algorithms assume that each vote for what is interesting and popular is independent, the choice of what votes are considered, and whether ballot-box stuffing is allowed, makes a huge difference in what people see. Humans make these choices. The algorithms have no agency. It is people, working in teams inside companies, that make choices on how to tune algorithms and what data is used by wisdom of the crowd algorithms. Those people can choose to do things one way, or they can choose to do them another way.

“Facebook employees decide what data sources the software can draw on in making its predictions,” reported the Washington Post. “And they decide what its goals should be — that is, what measurable outcomes to maximize for, and the relative importance of each.”

Small choices by teams inside of these companies can make a big difference for what the algorithms do. “Depending on the lever, the effects of even a tiny tweak can ripple across the network,” wrote the Washington Post in another article titled “Five Points for Anger, One Point for Like”. People control the algorithms. By tuning the algorithms, teams inside Facebook are “shaping whether the news sources in your feed are reputable or sketchy, political or not, whether you saw more of your real friends or more posts from groups Facebook wanted you to join, or if what you saw would be likely to anger, bore or inspire you.”

It's hard to find the right solutions if you don't first correctly identify the problem. The problem is not the algorithm. The problem is how people optimize the algorithm. People control what the algorithms do. What wisdom of the crowd algorithms choose to show depends on the incentives people have.

(This was an excerpt from the draft of my book. Please let me know if you like it and want more.)

Book excerpt: Metrics chasing engagement

(This is an excerpt from the draft of my book. Please let me know if you like it and want more.)

Let’s say you are in charge of building a social media website like Facebook. And you want to give your teams a goal, a target, some way to measure that what they are about to launch on the website is better than what came before.

One metric you might think of might be how much people engage with the website. You might think, every click, every like, every share, you can measure those. The more the better! We want people clicking, liking, and sharing as much as possible. Right?

So you tell all your teams, get people clicking! The more likes the better! Let’s go!

Teams are always looking for ways to optimize the metrics. Teams are constantly changing algorithms. If you tell your teams to optimize for clicks, what you will see is that soon recommender and ranker algorithms will change what they show. Up at the top of any recommendations and search results will be the posts and news predicted to get the most clicks.

Outside of the company, people will also notice and change what they do. They will say, this article I posted didn’t get much attention. But this one, wow, everyone clicked on it and reshared it. And people will create more of whatever does well on your site with the changes your team made to your algorithms.

All sounds great, right? What could go wrong?

The problem is what attracts the most clicks. What you are likely to click on are things that provoke strong emotions, such as hatred, disbelief, anger, or lust. This means what gets the most clicks are things that are lies, sensationalistic, provoking, or pornographic. The truth is boring. Posts of your Aunt Mildred’s flowers might make you happy. But they won’t get a click. But, oh yeah, that post with scurrilous lies about some dastardly other, that likely will get engagement.

Cecilia Kang and Sheera Frenkel wrote a book about Facebook, An Ugly Truth. In it, they describe the problem with how Facebook optimized its algorithms: “Over the years, the platform’s algorithms had gotten more sophisticated at identifying the material that appealed most to individual users and were prioritizing it at the top of their feeds. The News Feed operated like a finely tuned dial, sensitive to that photograph a user lingered on longest, or the article they spent the most time reading. Once it had established that the user was more likely to view a certain type of content, it fed them as much of it as possible.”

The content the algorithms fed to people, the content the algorithms chose to put on top and amplify, was not what made people content and satisfied. It was whatever would provide a click right now. And what would provide a click right now was often enraging lies.

“Engagement was 50 percent higher than in 2018 and 10 percent higher than in 2017,” wrote the author of the book The Hype Machine. “Each piece of content is scored according to our probabilities of engaging with it, across the several dozen engagement measures. Those engagement probabilities are aggregated into a single relevance score. Once the content is individually scored (Facebook’s algorithm considers about two thousand pieces of content for you every time you open your newsfeed), it’s ranked and shown in your feed in order of decreasing relevance.”

Most people will not read past the top few items in search results or on recommendations. So what is at the top is what matters most. In this case, by scoring and ordering content by likelihood of engagement, the content being amplified was the most sensationalistic content.

Once bad actors outside of Facebook discovered the weaknesses of the metrics behind the algorithms, they exploited it. From an article titled “Troll Farms Reached 140M Americans,” these are “easily exploited engagement based ranking systems … At the heart of Feed ranking, there are models that predict the probability a user will take an engagement action. These are colloquially known as P(like), P(comment), and P(share).” That is, the models use a prediction of the probability that people will like the content, the probability that they will share it, and so forth. Hao cited an internal report from Facebook that said that these “models heavily skew toward content we know to be bad.” Bad content includes hate speech, lies, and plagiarized content.

“Bad actors have learned how to easily exploit the systems,” said former Facebook data scientist Jeff Allen. “Basically, whatever score a piece of content got in the models when it was originally posted, it will likely get a similar score the second time it is posted … Bad actors can scrape … and repost … to watch it go viral all over again.” Anger, outrage, lies, and hate, all of those performed better on engagement metrics. They don’t make people satisfied. They make people more likely to leave in disgust than keep coming back. But they do make people likely to click right now.

It is by no means necessary to optimize for short-term engagement. Sarah Frier in the book No Filter describes how Instagram, in its early years, looked at what was happening at Facebook and made a different choice: “They decided the algorithm wouldn’t be formulated like the Facebook news feed, which had a goal of getting people to spend more time on Facebook … They knew where that road had led Facebook. Facebook had evolved into a mire of clickbait … whose presence exacerbated the problem of making regular people feel like they didn’t need to post. Instead Instagram trained the program to optimize for ‘number of posts made.’ The new Instagram algorithm would show people whatever posts would inspire them to create more posts.” While optimizing for the number of posts made also could have bad incentives, such as encouraging spamming, most important is considering the incentives created by the metrics you pick and questioning whether your current metrics are the best thing for the long-term of your business.

YouTube is an example of a company that picked problematic metrics years ago, but then questioned what was happening, noticed the problem, and then fixed their metrics in recent years. While researchers noted problems with YouTube’s recommender system amplifying terrible content many years ago, in recent years they have mostly concluded that YouTube no longer algorithmically amplifies — though they do still host — hate speech and other harmful content.

The problem started, as described by the authors of the book System Error, when a Vice President at YouTube “wrote an email to the YouTube executive team arguing that ‘watch time, and only watch time’ should be the objective to improve at YouTube … He equated watch time with user happiness: if a person spends hours a day watching videos on YouTube, it must reveal a preference for engaging in that activity.” The executive went on to claim, “When users spend more of their valuable time watching YouTube videos, they must perforce be happier with those videos.”

It is important to realize that YouTube is a giant optimization machine, with teams and systems targeting whatever metric it is given to maximize that metric. In the paper “Deep Neural Networks for YouTube Recommendations,” YouTube researchers describe it: “YouTube represents one of the largest scale and most sophisticated industrial recommendation systems in existence … In a live experiment, we can measure subtle changes in click-through rate, watch time, and many other metrics that measure user engagement … Our goal is to predict expected watch time given training examples that are either positive (the video impression was clicked) or negative (the impression was not clicked).”

The problem is that optimizing your recommendation algorithm for immediate watch time, which is an engagement metric, tends to show sensationalistic, scammy, and extreme content, including hate speech. As BuzzFeed reporters wrote in an article titled “We Followed YouTube’s Recommendation Algorithm Down the Rabbit Hole”: “YouTube users who turn to the platform for news and information — more than half of all users, according to the Pew Research Center — aren’t well served by its haphazard recommendation algorithm, which seems to be driven by an id that demands engagement above all else.”

The reporters described a particularly egregious case: “How many clicks through YouTube’s Up Next recommendations does it take to go from an anodyne PBS clip about the 116th United States Congress to an anti-immigrant video from a designated hate organization? Thanks to the site’s recommendation algorithm, just nine.” But the problem was not isolated to just a small number of examples. At the time, there were a “high percentage of users who say they’ve accepted suggestions from the Up Next algorithm — 81%.” The problem is that the optimization engines for their recommender algorithms. “It’s an engagement monster.”

The “algorithm decided which videos YouTube recommended that users watch next; the company said it was responsible for 70 percent of the one billion hours a day people spent on YouTube. But it had become clear that those recommendations tended to steer viewers toward videos that were hyperpartisan, divisive, misleading or downright false.” The problem was optimizing for an engagement metric like watch time.

Why does this happen? In any company, in any organization, you get what you measure. When you tell your teams to optimize for a certain metric, that they will get bonuses and be promoted if they optimize for that metric, they will optimize the hell out of that metric. As Bloomberg reporters wrote in an article titled “YouTube Executives Ignored Warnings,” “Product tells us that we want to increase this metric, then we go and increase it … Company managers failed to appreciate how [it] could backfire … The more outrageous the content, the more views.”

This problem was made substantially worse at YouTube by outright manipulation of YouTube’s wisdom of the crowd algorithms by adversaries, who effectively stuffed the ballot box for what is popular and good with votes from fake or controlled accounts. As Guardian reporters wrote, “Videos were clearly boosted by a vigorous, sustained social media campaign involving thousands of accounts controlled by political operatives, including a large number of bots … clear evidence of coordinated manipulation.”

The algorithms optimized for engagement, but they were perfectly happy to optimize for fake engagement, clicks and views from accounts that were all controlled by a small number of people. By pretending to be a large number of people, adversaries easily could make whatever they want appear popular, and also then get it amplified by a recommender algorithm that was greedy for more engagement.

In a later article, “Fiction is Outperforming Reality,” Paul Lewis at the Guardian wrote, “YouTube was six times more likely to recommend videos that aided Trump than his adversary. YouTube presumably never programmed its algorithm to benefit one candidate over another. But based on this evidence, at least, that is exactly what happened … Many of the videos appeared to have been pushed by networks of Twitter sock puppets and bots.” That is, Trump videos were not actually better to recommend, but manipulation by bad actors using a network of fake and controlled accounts caused the recommender to believe that it should recommend those videos. Ultimately, the metrics they picked, metrics that emphasized immediate engagement rather than the long-term, were at fault.

“YouTube’s recommendation system has probably figured out that edgy and hateful content is engaging.” As sociologist Zeynep Tufekci described it, “This is a bit like an autopilot cafeteria in a school that has figured out children have sweet teeth, and also like fatty and salty foods. So you make a line offering such food, automatically loading the next plate as soon as the bag of chips or candy in front of the young person has been consumed.” If the target of the optimization of the algorithms is engagement, the algorithms will be changed over time to automatically show the most engaging content, whether it contains useful information or full of lies and anger.

The algorithms were “leading people down hateful rabbit holes full of misinformation and lies at scale.” Why? “Because it works to increase the time people spend on the site” watching videos.

Later, YouTube stopped optimizing for watch time, but only years after seeing how much harmful content was recommended by YouTube algorithms. At the time, chasing engagement metrics changed both what people watched on YouTube and what videos got produced for YouTube. As one YouTube creator said, “We learned to fuel it and do whatever it took to please the algorithm.” Whatever metrics the algorithm was optimizing for, they did whatever it takes to please it. Pick the wrong metrics and the wrong things will happen, for customers and for the business.

(This was an excerpt from the draft of my book. Please let me know if you like it and want more.)