All Research Areas
Research Areas
Year Published

112 Results

December 23, 2011

High-efficiency server design

ACM Conference on Supercomputing (ICS)

Large-scale datacenters consume megawatts in power and cost hundreds of millions of dollars to equip. Reducing the energy and cost footprint of servers can therefore have substantial impact.

By: Eitan Frachtenberg, Ali Heydari, Harry Li, Amir Michael, Jacob Na, Avery Nisbet, Pierluigi Sarti
July 1, 2011

Many-core key-value store

International Green Computing Conference (IGCC)

Scaling data centers to handle task-parallel workloads requires balancing the cost of hardware, operations, and power. Low-power, low-core-count servers reduce costs in one of these dimensions, but may require additional nodes to provide the required quality of service or increase costs by underutilizing memory and other resources.

By: Mateusz Berezecki, Eitan Frachtenberg, Michael Paleczny, Ken Steele
June 20, 2011

YSmart: Yet Another SQL-to-MapReduce Translator

International Conference on Distributed Computing Systems (ICDCS)

MapReduce has become an effective approach to big data analytics in large cluster systems, where SQL-like queries play important roles to interface between users and systems. However, based on our Face book daily operation results, certain types of queries are executed at an unacceptable low speed by Hive (a production SQL-to-MapReduce translator). In this paper, we demonstrate that existing SQL-to-MapReduce translators that operate in a one-operation-to-one-job mode and do not consider query correlations cannot generate high-performance MapReduce programs for certain queries, due to the mismatch between complex SQL structures and simple MapReduce framework. We propose and develop a system called Y Smart, a correlation aware SQL-to-MapReduce translator. Y Smart applies a set of rules to use the minimal number of MapReduce jobs to execute multiple correlated operations in a complex query. Y Smart can significantly reduce redundant computations, I/O operations and network transfers compared to existing translators. We have implemented Y Smart with intensive evaluation for complex queries on two Amazon EC2 clusters and one Face book production cluster. The results show that Y Smart can outperform Hive and Pig, two widely used SQL-to-MapReduce translators, by more than four times for query execution.

By: Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongqiang He, Xiaodong Zhang
June 12, 2011

Apache Hadoop goes realtime at Facebook

ACM Special Interest Group on Management of Data (SIGMOD)

Facebook recently deployed Facebook Messages, its first ever user-facing application built on the Apache Hadoop platform. Apache HBase is a database-like layer built on Hadoop designed to support billions of messages per day.

By: Dhruba Borthakur, Joydeep Sen Sarma, Jonathan Gray, Kannan Muthukkaruppan, Nicolas Spiegelberg, Hairong Kuang, Karthik Ranganathan, Dmytro Molkov, Aravind Menon, Samuel Rash, Rodrigo Schmidt, Amitanand Aiyer
April 10, 2011

Facebook Immune System

Workshop on Social Network Systems (SNS)

Popular Internet sites are under attack all the time from phishers, fraudsters, and spammers. They aim to steal user information and expose users to unwanted spam. The attackers have vast resources at their disposal. They are well-funded, with full-time skilled labor, control over compromised and infected accounts, and access to global botnets.

By: Tao Stein, Roger Chen, Karan Mangla
March 30, 2011

FATE and DESTINI: A Framework for Cloud Recovery Testing

USENIX Symposium on Networked Systems Design and Implementation (NSDI)

As the cloud era begins and failures become commonplace, the fate and destiny of availability, reliability and performance are in the hands of failure recovery. Unfortunately, recovery problems still take place, causing downtimes, data loss, and many other problems.

By: Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Peter Alvaro, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen, Dhruba Borthakur
January 1, 2011

RCFile: A Fast and Space-efficient Data Placement Structure in MapReduce-based Warehouse Systems

IEEE International Conference on Data Engineering (ICDE)

MapReduce-based data warehouse systems are playing important roles of supporting big data analytics to understand quickly the dynamics of user behavior trends and their needs in typical Web service providers and social network sites (e.g., Facebook). In such a system, the data placement structure is a critical factor that can affect the warehouse performance in a fundamental way.

By: Yongqiang He, Rubao Lee, Yin Huai, Zheng Shao, Namit Jain, Xiaodong Zhang, Zhiwei Xu
October 4, 2010

Finding a needle in Haystack: Facebook’s photo storage

USENIX Symposium on Operating Systems Design and Implementation (OSDI)

This paper describes Haystack, an object storage system optimized for Facebook’s Photos application. Facebook currently stores over 260 billion images, which translates to over 20 petabytes of data. U…

By: Doug Beaver, Sanjeev Kumar, Harry Li, Jason Sobel, Peter Vajgel
June 6, 2010

Data warehousing and analytics infrastructure at Facebook.

Special Interest Group on Management of Data (SIGMOD)

Scalable analysis on large data sets has been core to the functions of a number of teams at Facebook – both engineering and non-engineering. Apart from ad hoc analysis of data and creation of business intelligence dashboards by analysts across the company, a number of Facebook’s site features are also based on analyzing large data sets.

By: Ashish Thusoo, Dhruba Borthakur, Raghotham Murthy, Zheng Shao, Namit Jain, Hao Liu, Suresh Antony, Joydeep Sen Sarma
April 13, 2010

Job Scheduling for Multi-User MapReduce Clusters

ACM European Conference on Computer Systems (EUROSYS)

Sharing a MapReduce cluster between users is attractive because it enables statistical multiplexing (lowering costs) and allows users to share a common large data set. However, we find that traditiona…

By: Matei Zaharia, Dhruba Borthakur, Joydeep Sen Sarma, Khaled Elmeleegy, Scott Shenker, Ion Stoica