August 11, 2013
Uncertainty in Online Experiments with Dependent Data: An Evaluation of Bootstrap Methods
ACM Conference on Knowledge Discovery and Data Mining (KDD)
Many online experiments exhibit dependence between users and items. For example, in online advertising, observations that have a user or an ad in common are likely to be associated. Because of this, even in experiments involving millions of subjects, the difference in mean outcomes between control and treatment conditions can have substantial variance. Previous theoretical and simulation results demonstrate that not accounting for this kind of dependence structure can result in confidence intervals that are too narrow, leading to inaccurate hypothesis tests.
By: Eytan Bakshy, Dean Eckles