June 22, 2015

Revisiting Memory Errors in Large-Scale Production Data Centers: Analysis and Modeling of New Trends from the Field

IEEE/IFIP International Conference on Dependable Systems and Networks

In this paper, we analyze the memory errors in the entire fleet of servers at Facebook over the course of fourteen months, representing billions of device days.

Justin Meza, Qiang Wu, Sanjeev Kumar, Onur Mutlu
June 15, 2015

A Large-Scale Study of Flash Memory Failures in the Field

ACM Sigmetrics 2015

This paper presents the first large-scale study of flash-based SSD reliability in the field.

Justin Meza, Qiang Wu, Sanjeev Kumar, Onur Mutlu
May 19, 2015

Challenges to Adopting Stronger Consistency at Scale

Workshop on Hot Topics in Operating Systems

There have been many recent advances in distributed systems that provide stronger semantics for geo-replicated data stores like those underlying Facebook. At Facebook we are excited by these lines of research, but fundamental and operational challenges currently make it infeasible to incorporate these advances into deployed systems. This paper describes some of these challenges with the hope that future advances will address them.

Philippe Ajoux, Nathan Bronson, Sanjeev Kumar, Wyatt Lloyd, Kaushik Veeraraghavan
May 6, 2015

Wormhole: Reliable Pub-Sub to Support Geo-replicated Internet Services

12th USENIX Symposium on Networked Systems Design and Implementation

Wormhole is a publish-subscribe (pub-sub) system developed for use within Facebook’s geographically replicated datacenters. It is used to reliably replicate changes among several Facebook services including TAO, Graph Search and Memcache. This paper describes the design and implementation of Wormhole as well as the operational challenges of scaling the system to support the multiple data storage systems deployed at Facebook.

Yogeshwer Sharma, Philippe Ajoux, Petchean Ang, David Callies, Abhishek Choudhary, Laurent Demailly, Thomas Fersch, Liat Atsmon, Andrzej Kotulski, Sachin Kulkarni, Sanjeev Kumar, Hu Li, Jun Li, Evgeniy Makeev, Kowshik Prakasam, Robbert van Renesse, Sabyasachi Roy, Pratyush Seth, Yee Jiun Song, Kaushik Veeraraghavan, Benjamin Wester, Peter Xie
February 10, 2015

Moving Fast with Software Verification

NASA Formal Method Symposium

For organisations like Facebook, high quality software is important. However, the pace of change and increasing complexity of modern code makes it difficult to produce error free software. Available tools are often lacking in helping programmers develop more reliable and secure applications.

Cristiano Calcagno, Dino Distefano, Jeremy Dubreil, Dominik Gabi, Pieter Hooimeijer, Martino Luca, Peter O'Hearn, Irene Papakonstantinou, Jim Purbrick, Dulma Churchill
October 27, 2014

Characterizing Load Imbalance in Real-World Networked Caches

HotNets 2014: Thirteenth ACM Workshop on Hot Topics in Networks

Modern Web services rely extensively upon a tier of in-memory caches to reduce request latencies and alleviate load on backend servers. Within a given cache, items are typically partitioned across cache servers via consistent hashing, with the goal of balancing the number of items maintained by each cache server. Effects of consistent hashing vary by associated hashing function and partitioning ratio. Most real-world workloads are also skewed, with some items significantly more popular than others. Inefficiency in addressing both issues can create an imbalance in cache-server loads. We analyze the degree of observed load imbalance, focusing on read-only traffic against Facebook’s graph cache tier in Tao. We investigate the principal causes of load imbalance, including data co-location, non-ideal hashing scenarios, and hot-spot temporal effects. We also employ trace-drive analytics to study the benefits and limitations of current load-balancing methods, suggesting areas for future research.

Qi Huang, Helga Gudmundsdottir, Ymir Vigfusson, Daniel A. Freedman, Ken Birman, Robbert van Renesse
October 24, 2014

The HipHop Virtual Machine

ACM International Conference on Object Oriented Programming Systems, Languages, and Applications

The HipHop Virtual Machine (HHVM) is a JIT compiler and runtime for PHP. While PHP values are dynamically typed, real programs often have latent types that are useful for optimization once discovered….

Keith Adams, Jason Evans, Bertrand Maher, Guilherme Ottoni, Drew Paroski, Brett Simmers, Edwin Smith, Owen Yamauchi
October 7, 2014

f4: Facebook’s Warm BLOB Storage System

Operating Systems Design and Implementation

Facebook’s corpus of photos, videos, and other Binary Large OBjects (BLOBs) that need to be reliably stored and quickly accessible is massive and continues to grow.

Subramanian Muralidhar, Wyatt Lloyd, Sabyasachi Roy, Cory Hill, Ernest Lin, Weiwen Liu, Satadru Pan, Shiva Shankar, Viswanath Sivakumar, Linpeng Tang, Sanjeev Kumar
October 6, 2014

The Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services

Operating Systems Design and Implementation

Current debugging and optimization methods scale poorly to deal with the complexity of modern Internet services, in which a single request triggers parallel execution of numerous heterogeneous softwar…

Mike Chow, David Meisner, Jason Flinn, Daniel Peek, Thomas Wenisch
September 18, 2014

Mining Energy Traces to Aid in Software Development: An Empirical Case Study

ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

With the advent of increased computing on mobile devices such as phones and tablets, it has become crucial to pay attention to the energy consumption of mobile applications.

Ashish Gupta, Thomas Zimmermann, Christian Bird, Nachiappan Nagappan, Thirumalesh Bhat, Syed Emran
August 24, 2014

Streamed Approximate Counting of Distinct Elements

ACM Conference on Knowledge Discovery and Data Mining (KDD)

Counting the number of distinct elements in a large dataset is a common task in web applications and databases. This problem is difficult in limited memory settings where storing a large hash table ta…

Daniel Ting
August 18, 2014

A Hitchhiker’s Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers

ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM)

Erasure codes such as Reed-Solomon (RS) codes are being extensively deployed in data centers since they offer significantly higher reliability than data replication methods at much lower storage overheads. These codes however mandate much higher resources with respect to network bandwidth and disk IO during reconstruction of data that is missing or otherwise unavailable.

K.V. Rashmi, Nihar B. Shah, Dikang Gu, Hairong Kuang, Dhruba Borthakur, Kannan Ramchandran
August 18, 2014

Fastpass: A Centralized “Zero-Queue” Datacenter Network

ACM Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM)

Current datacenter networks inherit the principles that went into the design of the Internet, where packet transmission and path selection decisions are distributed among the endpoints and routers. In…

Jonathan Perry, Amy Ousterhout, Hari Balakrishnan, Devavrat Shah, Hans Fugal
June 24, 2014

Fast Database Restarts at Facebook

ACM Special Interest Group on Management of Data (SIGMOD)

Facebook engineers query multiple databases to monitor and analyze Facebook products and services. The fastest of these databases is Scuba, which achieves subsecond query response time by storing all…

Aakash Goel, Bhuwan Chopra, Ciprian Gerea, Dhrúv Mátáni, Josh Metzler, Fahim Ul Haq, Janet Wiener
June 23, 2014

Adaptive HTER Estimation for Document-Specific MT Post-Editing

Annual Conference of Association of Computational Linguistics (ACL)

We present an adaptive translation quality estimation (QE) method to predict the human targeted translation error rate (HTER) for a document-specific machine translation model. We first introduce fea…

Fei Huang, Jianming Xu, Abraham Ittycheriah, Salim Roukos
June 20, 2014

vCacheShare: Automated Server Flash Cache Space Management in a Virtualization Environment

USENIX Annual Technical Conference (ATC)

Server Flash Cache (SFC) is being increasingly adopted in virtualization environments for IO acceleration. Deciding the optimal SFC allocation among VMs or VM disks is a major pain-point, dominantly handled manually by administrators.

Fei Meng, Li Zhou, Xiaosong Ma, Sandeep Uttamchandani, Deng Liu
June 12, 2014

There is no Fork: an Abstraction for Efficient, Concurrent, and Concise Data Access

ACM SIGPLAN International Conference on Functional Programming (ICFP)

We describe a new programming idiom for concurrency, based on Applicative Functors, where concurrency is implicit in the Applicative <*> operator. The result is that concurrent programs can be written in a natural applicative style, and they retain a high degree of clarity and modularity while executing with maximal concurrency.

Simon Marlow, Louis Brandy, Jon Coens, Jon Purdy
April 7, 2014

Deduplicating a Places Database

International World Wide Web Conference (WWW)

We consider the problem of resolving duplicates in a database of places, where a place is defined as any entity that has a name and a physical location. When other auxiliary attributes like phone and full address are not available, deduplication based solely on names and approximate location becomes an extremely challenging problem that requires both domain knowledge as well an local geographical knowledge.

Philip Bohannon, Nilesh Dalvi, Marian Olteanu, Manish Raghavan
April 2, 2014

Libra: Divide and Conquer to Verify Forwarding Tables in Huge Networks

USENIX Symposium on Networked Systems Design and Implementation (NSDI)

Data center networks often have errors in the forwarding tables, causing packets to loop indefinitely, fall into black-holes or simply get dropped before they reach the correct destination. Finding fo…

James Hongyi Zeng, Shidong Zhang, Fei Ye, Vimal Kumar, Mickey Ju, Junda Liu, Nick McKeown, Amin Vahdat
February 17, 2014

Analysis of HDFS Under HBase: A Facebook Messages Case Study

USENIX Conference on File Storage Technologies (FAST)

We present a multilayer study of the Facebook Messages stack, which is based on HBase and HDFS. We collect and analyze HDFS traces to identify potential improvements, which we then evaluate via simulation.

Tyler Harter, Dhruba Borthakur, Siying Dong, Amitanand Aiyer, Liyin Tang, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau