Book excerpt: The irresistible lure of an unlocked house

(This is an excerpt from drafts of my unpublished book, "Algorithms and Misinformation: Why Wisdom of the Crowds Failed the Internet and How to Fix It")

Bad incentives and bad metrics create an opportunity. They are what allow bad guys to come in and take root. Scammers and propagandists can take advantage of poorly optimized algorithms to make algorithms promote whatever misinformation they like.

Adversaries outside of these companies see wisdom of the crowd algorithms as an opportunity for free advertising. By manipulating algorithms with fake crowds, such as an astroturf campaign of controlled accounts and bots pretending to be real people, bad actors can feign popularity. Wisdom of the crowds summarizes opinions of the crowd. If the crowd is full of shills, the opinions will be skewed in whatever direction the shills like.

There is a massive underground economy around purchasing five star reviews on Amazon — as well as offering one star reviews for competing products — that allows counterfeiters and fraudsters to purchase whatever reputation they like for questionable and even dangerous products. Third-party merchants selling counterfeit, fraudulent, or other illicit goods with very high profit margins buy reviews from these services, feigning high quality to unwitting Amazon customers. If they are caught, they simply create a new account, list all their items again, and buy more fake reviews.

Get-rich-quick scammers and questionable vitamin supplement dealers can buy fake crowds of bogus accounts on social media that like and share their false promises. Buying fake crowds of followers on social media that like and share your content is a mature service now with dealers offering access to thousands of accounts for a few hundred dollars. Scammers rely on these fake crowds shilling their wares to fool algorithms into promoting their scams.

Foreign operatives have buildings full of people, each employee sitting at a desk pretending to be hundreds of Americans at once. They spend long days at work on social media with their multitude of fake accounts, commenting, liking, following, and sharing, all with the goal of pushing their disinformation and propaganda. The propaganda effort was so successful that, by 2019, some of the largest pages on social media were controlled by foreign governments with interests not aligned with the United States. Using their multitude of fake accounts, they were able to fool social media algorithms into recommending their pages and posts. Hundreds of millions of Americans saw their propaganda.

It is cheap to buy fake crowds and swamp wisdom of the crowd algorithms with bogus data about what is popular. When the crowd isn’t real, the algorithms don’t work. Wisdom of the crowd relies on crowds of independent, real people. Fake crowds full of shills means there is no wisdom in that crowd.

When algorithms amplify scams and disinformation, it may increase a platform’s engagement metrics for the moment. But, in the long-run, the bad actors win and the company loses. It is easy for people inside of tech companies to unwittingly optimize their algorithms in ways that help scammers and propagandists and hurt customers.