People and Cookies: Imperfect Treatment Assignment in Online Experiments

WWW 2016


Identifying the same internet user across devices or over time is often infeasible. This presents a problem for online experiments, as it precludes person-level randomization. Randomization must instead be done using imperfect proxies for people, like cookies, email addresses or device identifiers. Cookies present a unique problem for randomized experiments because a user may be associated with multiple cookies, some of which might be assigned to the test group and some to the control group during an experiment, making inference at the person level difficult. We find the cookie treatment effect estimator converges to a weighted average of the marginal effects of treating more of a user’s cookies. If the marginal effects of cookie treatment exposure are positive and constant, it underestimates the person level treatment effect by a factor equal to the number of cookies per user. Using cookie assignment data from Atlas and advertising exposure and purchase data from Facebook, we compare simulated cookie and person level advertising effectiveness experiments. The effect on statistical power is substantial: we find that sample sizes in a cookie test need to be two to three times larger to achieve the same power as an experiment with perfect treatment assignment.

Related Publications

All Publications

The Determinants of Social Connectedness in Europe

Michael Bailey, Drew Johnston, Theresa Kuchler, Dominic Russel, Bogdan State, Johannes Stroebel

SocInfo - September 22, 2020

Robust Market Equilibria with Uncertain Preferences

Riley Murray, Christian Kroer, Alex Peysakhovich, Parikshit Shah

AAAI - February 12, 2020

Bandwidth-Optimized Parallel Algorithms for Sparse Matrix-Matrix Multiplication using Propagation Blocking

Zhixiang Gu, Jose Moreira, David Edelsohn, Ariful Azad

SPAA - July 1, 2020

Experimentation and Performance in Advertising: An Observational Survey of Firm Practices on Facebook

Julian Runge, Steve Geinitz, Simon Ejdemyr

Expert Systems with Applications - June 9, 2020

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy