Explore the latest research from Facebook

All Publications

December 8, 2020 Seungwhan Moon, Satwik Kottur, Paul A. Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Ahmad Beirami, Eunjoon Cho, Rajen Subba, Alborz Geramifard
Paper

Situated and Interactive Multimodal Conversations

Next generation virtual assistants are envisioned to handle multimodal inputs (e.g., vision, memories of previous interactions, and the user’s utterances), and perform multimodal actions (e.g., displaying a route while generating the system’s utterance). We introduce Situated Interactive MultiModal Conversations (SIMMC) as a new direction aimed at training agents that take multimodal actions grounded in a co-evolving multimodal input context in addition to the dialog history.
Paper
December 1, 2020 Breannan Smith, Chenglei Wu, He Wen, Patrick Peluse, Yaser Sheikh, Jessica Hodgins, Takaaki Shiratori
Paper

Constraining Dense Hand Surface Tracking with Elasticity

By extending recent advances in vision-based tracking and physically based animation, we present the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures, for both single hands and two hands.
Paper
November 25, 2020 Donglai Xiang, Fabian Prada, Chenglei Wu, Jessica Hodgins
Paper

MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video

We present a method to capture temporally coherent dynamic clothing deformation from a monocular RGB video input. In contrast to the existing literature, our method does not require a pre-scanned personalized mesh template, and thus can be applied to in-the-wild videos.
Paper
November 23, 2020 Mateusz Machalica, Alex Samylkin, Meredith Porth, Satish Chandra
Paper

Predictive Test Selection

Change-based testing is a key component of continuous integration at Facebook. However, a large number of tests coupled with a high rate of changes committed to our monolithic repository make it infeasible to run all potentially impacted tests on each change. We propose a new predictive test selection strategy which selects a subset of tests to exercise for each change submitted to the continuous integration system.
Paper
November 16, 2020 Changwon Jang, Olivier Mercier, Kiseung Bang, Gang Li, Yang Zhao, Douglas Lanman
Paper

Design and Fabrication of Freeform Holographic Optical Elements

We propose an optimization method for grating vector fields that accounts for the unique selectivity properties of HOEs. We further show how our pipeline can be applied to two distinct HOE fabrication methods.
Paper
November 9, 2020 Aakar Gupta, Majed Samad, Kenrick Kin, Per Ola Kristensson, Hrvoje Benko
Paper

Investigating Remote Tactile Feedback for Mid-Air Text-Entry in Virtual Reality

In this paper, we investigate the utility of remote tactile feedback for freehand text-entry on a mid-air Qwerty keyboard in VR. To that end, we use insights from prior work to design a virtual keyboard along with different forms of tactile feedback, both spatial and non-spatial, for fingers and for wrists.
Paper
October 26, 2020 Ravish Mehra, W. Owen Brimijoin, Philip Robinson, Thomas Lunner
Paper

Potential of Augmented Reality Platforms to Improve Individual Hearing Aids and to Support More Ecologically Valid Research

In this article, suggestions are made about why AR platforms may offer ideal affordances to compensate for hearing loss, and how research-focused AR platforms could help toward better understanding of the role of hearing in everyday life.
Paper
October 25, 2020 Hanan Beit-On, Vladimir Tourbabin, Boaz Rafaely
Paper

The importance of time-frequency averaging for binaural speaker localization in reverberant environments

A common approach to overcoming the effect of reverberation in speaker localization is to identify the time-frequency (TF) bins in which the direct path is dominant, and then to use only these bins for estimation. Various direct-path dominance (DPD) tests have been proposed for identifying the direct-path bins. However, for a two-microphone binaural array, tests that do not employ averaging over TF bins seem to fail. In this paper, this anomaly is studied by comparing two DPD tests, in which only one has been designed to employ averaging over TF bins.
Paper
October 25, 2020 Ran Weisman, Vladimir Tourbabin, Paul Calamia, Boaz Rafaely
Paper

Spatial Covariance Matrix Estimation for Reverberant Speech with Application to Speech Enhancement

In this paper, a method for estimating the SCM of reverberant speech is proposed, based on the selection of time-frequency bins dominated by reverberation. The method is data-based and estimates the SCM for a specific acoustic scene. It is therefore applicable to realistic reverberant fields.
Paper
October 23, 2020 Paul Calamia, Nava Balsam, Philip Robinson
Paper

Blind estimation of the direct-to-reverberant ratio using a beta distribution fit to binaural coherence

This paper describes a method for blind estimation of the DRR which involves fitting a beta distribution to the magnitude-squared coherence between two binaural audio signals, aggregated over time and frequency.
Paper