Design and Analysis of Benchmarking Experiments for Distributed Internet Services

Proc. WWW


The successful development and deployment of large-scale Inter- net services depends critically on performance. Even small regressions in processing time can translate directly into significant energy and user experience costs. Despite the widespread use of distributed server infrastructure (e.g., in cloud computing and Web services), there is little research on how to benchmark such systems to obtain valid and precise inferences with minimal data collection costs. Correctly A/B testing distributed Internet services can be surprisingly difficult because interdependencies between user requests (e.g., for search results, social media streams, photos) and host servers violate assumptions required by standard statistical tests.

We develop statistical models of distributed Internet service performance based on data from Perflab, a production system used at Facebook which vets thousands of changes to the company’s codebase each day. We show how these models can be used to understand the tradeoffs between different benchmarking routines, and what factors must be taken into account when performing statistical tests. Using simulations and empirical data from Perflab, we validate our theoretical results, and provide easy-to-implement guidelines for designing and analyzing such benchmarks.

Related Publications

All Publications

Journal of Big Data - July 19, 2021

Cumulative deviation of a subpopulation from the full population

Mark Tygert

CSCW - November 3, 2020

The dynamics of U.S. college network formation on Facebook

Jan Overgoor, Bogdan State, Lada Adamic

PLOS ONE Journal - September 8, 2021

Large-scale decrease in the social salience of climate change during the COVID-19 pandemic

Brian R. Spisak, Bogdan State, Ingrid van de Leemput, Marten Scheffer, Yuwei Liu

AISTATS - August 31, 2021

Causal Autoregressive Flows

Ilyes Khemakhem, Ricardo P. Monti, Robert Leech, Aapo Hyvärinen

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy