The first recommendation feature at Amazon.com, BookMatcher, did not work very well. It required 20+ ratings to make recommendations. The recommendations leaned heavily toward bestsellers. The recommendations were often uninteresting, spurious, or obvious. Adding insult to injury, the system was always in danger of falling down under load.
As another side project (see "Early Amazon: Inventory cache"), I started working on a replacement. I wanted the recommendation engine to be fast, scalable, and able to work from just a couple ratings or purchases.
This was not what I was supposed to be doing, mind you.
I never got a chance to work on the BookMatcher project. I wasn't supposed to be working on recommendations. There were other, official projects that consumed my normal waking hours.
This was something I did on the side. My twisted, geek idea of fun.
I hacked. I tested. I iterated. With time, I built something that worked well.
I showed it to a few other people. Right around then, there was a major redesign of the Amazon.com website. Before I knew it, I was rushing my little prototype out the door.
A talented engineering manager named Dwayne helped put a spiffy UI on top of the recommendation engine. We rolled it out with the other changes. The new feature was called "Instant Recommendations".
Bookmatcher and Instant Recommendations co-existed on the site for some time. Eventually, low traffic, performance problems, and poor recommendation quality caused BookMatcher to disappear.
Instant Recommendations expanded to take its place, and eventually formed the backbone for much of the personalization at Amazon.
Such a useful and valuable feature came from work I was never supposed to be doing. In there somewhere lies the secret to innovation.