All Research Areas
Research Areas
Year Published

56 Results

June 20, 2011

YSmart: Yet Another SQL-to-MapReduce Translator

International Conference on Distributed Computing Systems (ICDCS)

MapReduce has become an effective approach to big data analytics in large cluster systems, where SQL-like queries play important roles to interface between users and systems. However, based on our Face book daily operation results, certain types of queries are executed at an unacceptable low speed by Hive (a production SQL-to-MapReduce translator). In this paper, we demonstrate that existing SQL-to-MapReduce translators that operate in a one-operation-to-one-job mode and do not consider query correlations cannot generate high-performance MapReduce programs for certain queries, due to the mismatch between complex SQL structures and simple MapReduce framework. We propose and develop a system called Y Smart, a correlation aware SQL-to-MapReduce translator. Y Smart applies a set of rules to use the minimal number of MapReduce jobs to execute multiple correlated operations in a complex query. Y Smart can significantly reduce redundant computations, I/O operations and network transfers compared to existing translators. We have implemented Y Smart with intensive evaluation for complex queries on two Amazon EC2 clusters and one Face book production cluster. The results show that Y Smart can outperform Hive and Pig, two widely used SQL-to-MapReduce translators, by more than four times for query execution.

By: Rubao Lee, Tian Luo, Yin Huai, Fusheng Wang, Yongqiang He, Xiaodong Zhang
January 1, 2011

Supervised Random Walks: Predicting and Recommending Links in Social Networks

ACM International Conference on Web Search and Data Mining (WSDM)

Predicting the occurrence of links is a fundamental problem in networks. In the link prediction problem we are given a snapshot of a network and would like to infer which interactions among existing members are likely to occur in the near future or which existing interactions are we missing. Although this problem has been extensively studied, the challenge of how to effectively combine the information from the network structure with rich node and edge attribute data remains largely open.

By: Lars Backstrom, Jure Leskovec
June 1, 2010

Tools for Collecting Speech Corpora via Mechanical Turk

NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk

To rapidly port speech applications to new languages one of the most difficult tasks is the initial collection of sufficient speech corpora.

By: Ian Lane, Alex Waibel, Matthias Eck, Kay Rottmann
June 1, 2010

Not-so-latent dirichlet allocation: collapsed Gibbs sampling using human judgments

Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)

Probabilistic topic models are a popular tool for the unsupervised analysis of text, providing both a predictive model of future text and a latent topic representation of the corpus. Recent studies have found that while there are suggestive connections between topic models and the way humans interpret data, these two often disagree.

By: Jonathan Chang
April 19, 2010

ePluribus: Ethnicity on Social Networks


We propose an approach to determine the ethnic break-down of a population based solely on people’s names and data provided by the U.S. Census Bureau. We demonstrate that our approach is able to predict the ethnicities of individuals as well as the ethnicity of an entire population better than natural alternatives.

By: Jonathan Chang, Itamar Rosenn, Lars Backstrom, Cameron Marlow
February 24, 2018

Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective

International Symposium on High-Performance Computer Architecture (HPCA)

Facebook’s machine learning workloads are extremely diverse: services require many different types of models in practice. This paper describes the hardware and software infrastructure that supports machine learning at global scale.

By: Kim Hazelwood, Sarah Bird, David Brooks, Soumith Chintala, Utku Diril, Dmytro Dzhulgakov, Mohamed Fawzy, Bill Jia, Yangqing Jia, Aditya Kalro, James Law, Kevin Lee, Jason Lu, Pieter Noordhuis, Misha Smelyanskiy, Liang Xiong, Xiaodong Wang