tag:blogger.com,1999:blog-6569681.post4411071282159817027..comments2024-03-29T05:14:10.903-07:00Comments on Geeking with Greg: New Google study on speed in search resultsGreg Lindenhttp://www.blogger.com/profile/09216403000599463072noreply@blogger.comBlogger14125tag:blogger.com,1999:blog-6569681.post-3644985101905365332010-07-17T10:11:35.119-07:002010-07-17T10:11:35.119-07:00The experiment Marissa Mayer referenced changed th...The experiment Marissa Mayer referenced changed the default number of search results on the page. This is a different mechanism of increasing latency than injecting server-side delay (server-side delay underlies the results described in <a href="http://googleresearch.blogspot.com/2009/06/speed-matters.html" rel="nofollow">this blog post</a>). Furthermore, as fnthawar comments, it is not clear one can untangle the user response to additional results vs. additional latency to generate those results.Anonymoushttps://www.blogger.com/profile/12377818866132075147noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-52133708250632297032009-08-07T14:41:21.416-07:002009-08-07T14:41:21.416-07:00I bet the 20% drop in revenue was mostly because o...I bet the 20% drop in revenue was mostly because of fewer *ad clicks*, not fewer *searches*. Going to the next page of Google results brings up more ads. Scrolling down the page doesn't.randallhttps://www.blogger.com/profile/04270395955627579493noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-70209077259537607832009-07-14T10:12:33.378-07:002009-07-14T10:12:33.378-07:00Don't forget, correlation does not equal causa...Don't forget, correlation does not equal causation. The original Marissa Mayer study was trying out 20 search result links vs. 10. It also happened to be a slower loading page.<br /><br />Just because user's didn't like 20 links AND a slower page, doesn't mean the slowness was the only cause of less usage.fnthawarhttps://www.blogger.com/profile/00706919378319013894noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-33967456808199279762009-07-08T10:32:46.750-07:002009-07-08T10:32:46.750-07:00A talk by Eric and Jake @ Velocity, 09 -- http://b...A talk by Eric and Jake @ Velocity, 09 -- http://blip.tv/file/2279751Shirishhttps://www.blogger.com/profile/12091474325051553432noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-61762615689638483622009-07-07T07:55:25.326-07:002009-07-07T07:55:25.326-07:00In a separate e-mail, the author of the paper, Jak...In a separate e-mail, the author of the paper, Jake Brutlag, suggests that the difference is likely due to confounding factors such as additional client side latency when adding 30 results.<br /><br />Jake also pointed to Marissa Mayer's <a href="http://blip.tv/file/2290442" rel="nofollow">Velocity 09 talk</a>, where (starting around 5:00) Marissa explicitly discusses these different results, though she doesn't go into why they differ by such a large amount.<br /><br />Given the magnitude of the difference, it is hard not to still have questions about these results, but perhaps that helps a little.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-34582607567537565912009-06-30T08:35:35.515-07:002009-06-30T08:35:35.515-07:00It looks like Jake has published his e-mail on his...It looks like Jake has published his e-mail on his home page. It is jakeb[at]google.com.<br /><br />I get what you and Ewout are saying, that this study not only appears to be inconsistent with previous reports on the magnitude of the effect, but wildly inconsistent. It would be interesting to get Jake's thoughts on that if he is willing to share them.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-554508763823772992009-06-30T08:25:29.240-07:002009-06-30T08:25:29.240-07:00Sure, we can write Jake. His email isn't list...Sure, we can write Jake. His email isn't listen in the paper -- do you have it? <br /><br />Another though, while I'm on it: Let's even say that these numbers are correct. Let's assume that the 400ms delay causes a 0.6% drop in the number of searches performed, and that drop is statistically significant. (I don't see significance values reported in the paper, but.. whatever. Let's assume that it is significant.)<br /><br />Translated into real numbers, that means that for every 500 queries that a user used to do, he or she now does 497? Or, given the statistic that the average searcher does 4 searches per day.. that means that once every 125 days, the user does one less search? That's once every 4.2 months, or approximately three times a year that the user does one less search, than they otherwise would have. ONE.<br /><br />I can see how, maybe, from the corporation's perspective, all those little tiny differences add up to a significant difference in revenue. Especially when you aggregate over hundreds of millions of users. <br /><br />But I'm looking at this from the perspective of the user. And I am having a hard time seeing how this really affects the user, at all. If once every four months I feel so inclined to not perform a search that I otherwise would have, I don't think that I would ever even notice that, or in any way be inconvenienced by that fact. <br /><br />I don't get it.<br /><br />And so it seems to me that a much better use of corporate resources would be to make sure that a larger percentage of the searches that a user does succeeds. What is the statistic.. something like 50% of all searches end in failure? If you could lower that failure rate to, say, 45%, then the end user would be much happier, than if you lower delay from 400ms to 100ms.jeremyhttp://irgupf.comnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-90395337910050385462009-06-30T08:14:42.182-07:002009-06-30T08:14:42.182-07:00That's a good point, Jeremy. I don't have...That's a good point, Jeremy. I don't have a good explanation for this inconsistency since my previous explanation breaks down if the -20% drop was also to searches. <br /><br />Perhaps might be worth writing Jake Brutlag to get his thoughts?Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-51381815995312028742009-06-30T07:59:57.614-07:002009-06-30T07:59:57.614-07:00By the way, you mentioned that you hadn't seen...<i>By the way, you mentioned that you hadn't seen Marissa give that +500ms leads to a -20% revenue drop data point?</i><br /><br />Greg,<br /><br />In your cited Nov 2006 post, you wrote: "<i>Traffic and revenue from Google searchers in the experimental group dropped by 20%.</i>" Not just revenue, but traffic. Which I assume is pretty much a 1:1 correspondence with number of searches issued.<br /><br />So I'm kinda with Ewout, in scratching my head in a little bit of confusion over this one. Let's see.. <br /><br />200 ms delay = 0.2% traffic drop (2009 study)<br />400 ms delay = 0.6% traffic drop (2009 study)<br />500 ms delay = 20.0% traffic drop (2006 study)<br /><br />I'd be hard pressed to fit any sort of regression line to that data. Does that additional 100 ms, when going from a 400ms to 500ms delay, really cause an additional 19.4% drop in usage?jeremyhttp://irgupf.comnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-52915433336159848272009-06-30T07:31:53.890-07:002009-06-30T07:31:53.890-07:00It's a good point, Ewout, that the data has wi...It's a good point, Ewout, that the data has wide variation.<br /><br />In part, I think this is because the experiments differed in how much and where they did the delay as well as what metric they used to determine impact.<br /><br />For example, if the search results appeared significantly before the ads appeared, you might expect to see a fraction of people click on the search results before the ads even display.<br /><br />In the end, how much speed impacts perceived quality in a particular application probably depends on the usage and interface design. What these studies indicate is that we probably should be concerned about speed. How much we should be concerned likely is something that would have to be determined for each application.<br /><br />By the way, you mentioned that you hadn't seen Marissa give that +500ms leads to a -20% revenue drop data point? If you want, you can see it in her "<a href="http://video.google.com/videoplay?docid=-7039469220993285507" rel="nofollow">Scaling Google for the Everyday User</a>" talk at the Seattle Scalability Conference in 2007 (see around 13:00 in the video).Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-5320236682168874492009-06-30T07:21:27.070-07:002009-06-30T07:21:27.070-07:00Thanks for the link! I noticed that the link had ...Thanks for the link! I noticed that the link had a typo (beter instead of better) so I just corrected that. If possible, please update your link. Thanks!Unknownhttps://www.blogger.com/profile/15129849152587392825noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-17214028560608585222009-06-30T04:09:21.371-07:002009-06-30T04:09:21.371-07:00Thanks for the Dries Buytaert reference.
I just ...Thanks for the Dries Buytaert reference. <br /><br />I just don't see it. The Marrissa Mayer data is now a meme (20%-30%/0.1 sec, see http://twitter.com/chabotc/status/2401905834 for example) but unlike you, I never saw real data. <br /><br />All of the other data, including yours (1% drop in sales / 0.1s delay, according to the Dries post), give much smaller effects.<br /><br />"Speed matters" is obviously true, but what is the real quantitative meaning behind the slogan? 10s of % effects/0.1sec is really different than 1% or 0.1%/0.1s and leads to different priorities.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-54461886994831483452009-06-29T16:17:05.729-07:002009-06-29T16:17:05.729-07:00No, I wouldn't conclude that.
The 2006 data f...No, I wouldn't conclude that.<br /><br />The 2006 data from Google was also the result of a rigorous A/B test. I wouldn't say this new test disqualifies the old.<br /><br />I also wouldn't agree that they are in conflict. They measured different things with different delays in different places.<br /><br />One measured revenue drops with a longer delay that may have occurred after the search results appeared and before the advertising rendered.<br /><br />The other measured usage drops with shorter delays that occurred before any content on the page rendered.<br /><br />I would take this second Google study as another data point, like you should the Amazon studies and Yahoo studies that also showed the impacts of delays on user satisfaction.Greg Lindenhttps://www.blogger.com/profile/09216403000599463072noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-40890385096095412332009-06-29T16:10:26.722-07:002009-06-29T16:10:26.722-07:00So if I understand correctly, the idea that the 50...So if I understand correctly, the idea that the 500ms delay in 2006 was the cause of the 20% drop in revenue (proportional to further use and searches, I guess) was completely dis-proofed by actual, rigorous testing.<br /><br />As I read these numbers, speed does not matter at all (even almost half a second of delay caused an almost negligible drop in use)<br /><br />According to these number, we are much better of looking for other factors like usability to improve our sites.Anonymousnoreply@blogger.com