Publication

Top Challenges from the first Practical Online Controlled Experiments Summit

SIGKDD Explorations


Abstract

Online controlled experiments (OCEs), also known as A/B tests, have become ubiquitous in evaluating the impact of changes made to software products and services. While the concept of online controlled experiments is simple, there are many practical challenges in running OCEs at scale and encourage further academic and industrial exploration. To understand the top practical challenges in running OCEs at scale, representatives with experience in large-scale experimentation from thirteen different organizations (Airbnb, Amazon, Booking.com, Facebook, Google, LinkedIn, Lyft, Microsoft, Netflix, Twitter, Uber, Yandex, and Stanford University) were invited to the first Practical Online Controlled Experiments Summit. All thirteen organizations sent representatives. Together these organizations tested more than one hundred thousand experiment treatments last year. Thirty-four experts from these organizations participated in the summit in Sunnyvale, CA, USA on December 13-14, 2018.

While there are papers from individual organizations on some of the challenges and pitfalls in running OCEs at scale, this is the first paper to provide the top challenges faced across the industry for running OCEs at scale and some common solutions.

Related Publications

All Publications

Journal of Big Data - July 19, 2021

Cumulative deviation of a subpopulation from the full population

Mark Tygert

CSCW - November 3, 2020

The dynamics of U.S. college network formation on Facebook

Jan Overgoor, Bogdan State, Lada Adamic

PLOS ONE Journal - September 8, 2021

Large-scale decrease in the social salience of climate change during the COVID-19 pandemic

Brian R. Spisak, Bogdan State, Ingrid van de Leemput, Marten Scheffer, Yuwei Liu

AISTATS - August 31, 2021

Causal Autoregressive Flows

Ilyes Khemakhem, Ricardo P. Monti, Robert Leech, Aapo Hyvärinen

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy