Variance-Weighted Estimators to Improve Sensitivity in Online Experiments

Conference on Economics and Computation (EC)


As companies increasingly rely on experiments to make product decisions, precisely measuring changes in key metrics is important. Various methods to increase sensitivity in experiments have been proposed, including methods that use pre-experiment data, machine learning, and more advanced experimental designs. However, prior work has not explored modeling heterogeneity in the variance of individual experimental users. We propose a more sensitive treatment effect estimator that relies on estimating the individual variances of experimental users using pre-experiment data. We show that that weighted estimators using individual-level variance estimates can reduce the variance of treatment effect estimates, and prove that the coefficient of variation of the sample population variance is a sufficient statistic for determining the scale of possible variance reduction. We provide empirical results from case studies at Facebook demonstrating the effectiveness of this approach, where the average experiment achieved a 17% reduction in variance with minimal impact on bias.

Related Publications

All Publications

Country Differences in Social Comparison on Social Media

Justin Cheng, Moira Burke, Bethany de Gant

CSCW - October 17, 2020

The Determinants of Social Connectedness in Europe

Michael Bailey, Drew Johnston, Theresa Kuchler, Dominic Russel, Bogdan State, Johannes Stroebel

SocInfo - September 22, 2020

Robust Market Equilibria with Uncertain Preferences

Riley Murray, Christian Kroer, Alex Peysakhovich, Parikshit Shah

AAAI - February 12, 2020

Bandwidth-Optimized Parallel Algorithms for Sparse Matrix-Matrix Multiplication using Propagation Blocking

Zhixiang Gu, Jose Moreira, David Edelsohn, Ariful Azad

SPAA - July 1, 2020

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy