People and Cookies: Imperfect Treatment Assignment in Online Experiments

WWW 2016


Identifying the same internet user across devices or over time is often infeasible. This presents a problem for online experiments, as it precludes person-level randomization. Randomization must instead be done using imperfect proxies for people, like cookies, email addresses or device identifiers. Cookies present a unique problem for randomized experiments because a user may be associated with multiple cookies, some of which might be assigned to the test group and some to the control group during an experiment, making inference at the person level difficult. We find the cookie treatment effect estimator converges to a weighted average of the marginal effects of treating more of a user’s cookies. If the marginal effects of cookie treatment exposure are positive and constant, it underestimates the person level treatment effect by a factor equal to the number of cookies per user. Using cookie assignment data from Atlas and advertising exposure and purchase data from Facebook, we compare simulated cookie and person level advertising effectiveness experiments. The effect on statistical power is substantial: we find that sample sizes in a cookie test need to be two to three times larger to achieve the same power as an experiment with perfect treatment assignment.

Related Publications

All Publications

Journal of Big Data - November 6, 2021

A graphical method of cumulative differences between two subpopulations

Mark Tygert

NeurIPS - December 6, 2021

Parallel Bayesian Optimization of Multiple Noisy Objectives with Expected Hypervolume Improvement

Samuel Daulton, Maximilian Balandat, Eytan Bakshy

KAIS - June 30, 2021

Fire Now, Fire Later: Alarm-Based Systems for Prescriptive Process Monitoring

Stephan A. Fahrenkrog-Petersen, Niek Tax, Irene Teinemaa, Marlon Dumas, Massimiliano de Leoni, Fabrizio Maria Maggi, Matthias Weidlich

The Quarterly Journal of Economics - October 25, 2021

Peer Effects in Product Adoption

Michael Bailey, Drew Johnston, Theresa Kuchler, Johannes Stroebel, Arlene Wong

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookie Policy