Book excerpt: From hope to despair and back to hope

(This is an excerpt from drafts of my book, "Algorithms and Misinformation: Why Wisdom of the Crowds Failed the Internet and How to Fix It")

Twenty five years ago, when recommendation algorithms first launched at large scale on the internet, these algorithms helped people discover new books to read and new movies to watch.

In recent years, wisdom of the crowds failed the internet, and the internet filled with misinformation.

The story of why this happened — and how to fix it — runs through the algorithms that pick what we see on the internet. Algorithms use wisdom of the crowds at a massive scale to find what is popular and interesting. That is how they determine what to show to millions of people. When these algorithms fail, misinformation flourishes.

The reason the algorithms fail is not what you think. It is not the algorithms.

Only with an insider view can readers see how the algorithms work and how tech companies build these algorithms. The surprise is that the algorithms are actually made of people.

People build and maintain these algorithms. Wisdom of the crowds works using data about what people do. The key to why algorithms go wrong, and how they can be fixed, runs through people and the incentives people have.

When bad actors manipulate algorithms, they are trying to get their scams and misinformation seen by as many people as possible as cheaply as possible.

When teams inside companies optimize algorithms, they are trying to meet the goals executives set for them, whatever those goals are and regardless of whether they are the right goals for the company.

People’s incentives control what the algorithms do. And incentives are the key to fixing misinformation on the internet.

To make wisdom of the crowds useful again, and to make misinformation ineffective, all companies must use only reliable data and must not optimize their algorithms for engagement. As this book shows, these solutions reduce the reach of misinformation, making it far less effective and far more expensive for scammers and fraudsters.

We know these solutions work because some companies did it. Exposing a gold mine of knowledge buried deep inside the major tech companies, this book shows that some successfully stopped their algorithms from amplifying misinformation by not optimizing for engagement. And, importantly, these companies made more money by doing so.

Companies that have not fixed their algorithms have taken a dark path, blinded by short-term optimization for engagement, and the teams deceived by bad incentives and bad metrics inside of their companies. This book shows the way out for those led astray.

People inside and outside the powerful tech companies, including consumers and policy makers, can help align incentives away from short-term engagement and toward long-term customer satisfaction and growth.

It turns out it's a win-win to listen to consumers and optimize algorithms to be helpful for your customers in the long-term. Nudging people's incentives in practical ways is easier once you see inside the companies, understand how they build these algorithms, and see that companies make more money when they do not myopically optimize their algorithms in ways that later will cause a flood of misinformation and scams.

Wisdom of the crowd algorithms are everywhere on the internet. Readers of this book started out feeling powerless to fix the algorithms that control everything we see and the misinformation these algorithms promote. Readers of this book end this book hopeful and ready to push for change.