From their help page:
The Website Optimizer allows you to test changes in the website content of your pages in order to determine what will be most effective in getting conversions.Much more detailed information in their technical overview.
You choose what parts of a page you'd like to test -- headline, image, promo text -- and we'll run an experiment on a portion of your site traffic to determine which content on your site users respond to best.
When we've collected enough data, we'll provide you with reliable reports and a suggested course of action in order to optimize your site for maximum business results.
It is no longer just Amazon and Google who have powerful A/B testing frameworks. Google appears to have just opened up their A/B testing tools to everyone.
[Found via Liam Morrison and Philipp Lenssen]