tag:blogger.com,1999:blog-6569681.post7097152757415790804..comments2024-01-15T13:17:33.771-08:00Comments on Geeking with Greg: Puzzling outcomes in A/B testingGreg Lindenhttp://www.blogger.com/profile/09216403000599463072noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-6569681.post-88595755166754057062012-07-30T17:10:59.072-07:002012-07-30T17:10:59.072-07:00You deserve a raise!You deserve a raise!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-8721281712667225592012-07-09T07:54:34.060-07:002012-07-09T07:54:34.060-07:00One question I've long had about A/B testing i...One question I've long had about A/B testing is how it holds up, not under different versioning, but under true system evolution. Not because you can't compare two different systems using the same metric (KPI/OEC). Rather, because what the system is doing or trying to do itself is changing, as it evolves. In other words, the metrics themselves need to change along with the system. But if you're metric isn't constant, then there is no hope of ever doing a valid scientific comparison, right?<br /><br />So what I'm asking is whether the "puzzling outcomes" questions that these authors are asking are really just manifestations of an underlying drift in the metric itself. A drift that the authors might not even be realizing that they subconsciously want the metric to make? <br /><br />This is a philosophical question rather than a technical one.jeremyhttp://irgupf.comnoreply@blogger.comtag:blogger.com,1999:blog-6569681.post-64006811210932517182012-07-06T23:57:55.026-07:002012-07-06T23:57:55.026-07:00It seems almost hopeless; even if your testing met...It seems almost hopeless; even if your testing methodology is unbiased, your subsequent analysis will almost certainly be. (Unexpected results will be analyzed until they make sense, and expected results will be taken almost at face value--it's hard to dig for a deeper explanation when a straightforward one is at hand.) Bummer.Jeff Cliteshttps://www.blogger.com/profile/16269464424219539990noreply@blogger.comtag:blogger.com,1999:blog-6569681.post-49055461203613872942012-07-06T16:29:41.042-07:002012-07-06T16:29:41.042-07:00Thanks for posting! Would you be able to share an...Thanks for posting! Would you be able to share any insight into the strategy and methodology used at Amazon for the long-term value measurement of users who saw pop-up ads? In my experience, it can be quite difficult to run long-term a/b experiments on a rapidly evolving web product.Anonymousnoreply@blogger.com