Variance-Weighted Estimators to Improve Sensitivity in Online Experiments

Conference on Economics and Computation (EC)


As companies increasingly rely on experiments to make product decisions, precisely measuring changes in key metrics is important. Various methods to increase sensitivity in experiments have been proposed, including methods that use pre-experiment data, machine learning, and more advanced experimental designs. However, prior work has not explored modeling heterogeneity in the variance of individual experimental users. We propose a more sensitive treatment effect estimator that relies on estimating the individual variances of experimental users using pre-experiment data. We show that that weighted estimators using individual-level variance estimates can reduce the variance of treatment effect estimates, and prove that the coefficient of variation of the sample population variance is a sufficient statistic for determining the scale of possible variance reduction. We provide empirical results from case studies at Facebook demonstrating the effectiveness of this approach, where the average experiment achieved a 17% reduction in variance with minimal impact on bias.

Related Publications

All Publications

NeurIPS - December 16, 2020

Online Bayesian Persuasion

Matteo Castiglioni, Andrea Celli, Alberto Marchesi, Nicola Gatti

WINE - December 7, 2020

The Ad Types Problem

Riccardo Colini Baldeschi, Julian Mestre, Okke Schrijvers, Christopher A. Wilkens

CODE - November 20, 2020

Privacy-Preserving Randomized Controlled Trials: A Protocol for Industry Scale Deployment (Extended Abstract)

Mahnush Movahedi, Benjamin M. Case, Andrew Knox, Li Li, Yiming Paul Li, Sanjay Saravanan, Shubho Sengupta, Erik Taubeneck

NBER - September 30, 2020

The Effects of COVID-19 on U.S. Small Businesses: Evidence from Owners, Managers, and Employees

Georgij Alekseev, Safaa Amer, Manasa Gopal, Theresa Kuchler, JW Schneider, Johannes Stroebel, Nils Wernerfelt

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy