Explore the latest research from Facebook

All Publications

December 1, 2020 Breannan Smith, Chenglei Wu, He Wen, Patrick Peluse, Yaser Sheikh, Jessica Hodgins, Takaaki Shiratori
Paper

Constraining Dense Hand Surface Tracking with Elasticity

By extending recent advances in vision-based tracking and physically based animation, we present the first algorithm capable of tracking high-fidelity hand deformations through highly self-contacting and self-occluding hand gestures, for both single hands and two hands.
Paper
October 26, 2020 Ravish Mehra, W. Owen Brimijoin, Philip Robinson, Thomas Lunner
Paper

Potential of Augmented Reality Platforms to Improve Individual Hearing Aids and to Support More Ecologically Valid Research

In this article, suggestions are made about why AR platforms may offer ideal affordances to compensate for hearing loss, and how research-focused AR platforms could help toward better understanding of the role of hearing in everyday life.
Paper
October 19, 2020 Jun Gong, Aakar Gupta, Hrvoje Benko
Paper

Acustico: Surface Tap Detection and Localization using Wrist-based Acoustic TDOA Sensing

In this paper, we present Acustico, a passive acoustic sensing approach that enables tap detection and 2D tap localization on uninstrumented surfaces using a wrist-worn device. Our technique uses a novel application of acoustic time differences of arrival (TDOA) analysis.
Paper
September 26, 2020 Edbert J. Sie, Hui Chen, E-Fann Saung, Ryan Catoen, Tobias Tiecke, Mark A. Chevillet, Francesco Marsili
Paper

High-sensitivity multispeckle diffuse correlation spectroscopy

Cerebral blood flow is an important biomarker of brain health and function as it regulates the delivery of oxygen and substrates to tissue and the removal of metabolic waste products. Moreover, blood flow changes in specific areas of the brain are correlated with neuronal activity in those areas. Diffuse correlation spectroscopy (DCS) is a promising noninvasive optical technique for monitoring cerebral blood flow and for measuring cortex functional activation tasks. However, the current state-of-the-art DCS adoption is hindered by a trade-off between sensitivity to the cortex and signal-to-noise ratio (SNR).
Paper
August 24, 2020 Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, Masayoshi Tomizuka
Paper

SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation

Using standard convolutions to process such LiDAR images is problematic, as convolution filters pick up local features that are only active in specific regions in the image. As a result, the capacity of the network is under-utilized and the segmentation performance decreases. To fix this, we propose Spatially-Adaptive Convolution (SAC) to adopt different filters for different locations according to the input image.
Paper
August 24, 2020 Hang Chu, Shugao Ma, Fernando De la Torre, Sanja Fidler, Yaser Sheikh
Paper

Expressive Telepresence via Modular Codec Avatars

VR telepresence consists of interacting with another human in a virtual space represented by an avatar. Today most avatars are cartoon-like, but soon the technology will allow video-realistic ones. This paper aims in this direction, and presents Modular Codec Avatars (MCA), a method to generate hyper-realistic faces driven by the cameras in the VR headset.
Paper
August 23, 2020 Gyeongsik Moon, Shoou-I Yu, He Wen, Takaaki Shiratori, Kyoung Mu Lee
Paper

InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image

Analysis of hand-hand interactions is a crucial step towards better understanding human behavior. However, most researches in 3D hand pose estimation have focused on the isolated single hand case. Therefore, we firstly propose (1) a large-scale dataset, InterHand2.6M, and (2) a baseline network, InterNet, for 3D interacting hand pose estimation from a single RGB image.
Paper
August 23, 2020 Samarth Brahmbhatt, Chengcheng Tang, Christopher D. Twigg, Charles C. Kemp, James Hays
Paper

ContactPose: A Dataset of Grasps with Object Contact and Hand Pose

We introduce ContactPose, the first dataset of hand-object contact paired with hand pose, object pose, and RGB-D images. ContactPose has 2306 unique grasps of 25 household objects grasped with 2 functional intents by 50 participants, and more than 2.9 M RGB-D grasp images. Analysis of ContactPose data reveals interesting relationships between hand pose and contact.
Paper
August 23, 2020 Gyeongsik Moon, Takaaki Shiratori, Kyoung Mu Lee
Paper

DeepHandMesh: A Weakly-Supervised Deep Encoder-Decoder Framework for High-Fidelity Hand Mesh Modeling

Human hands play a central role in interacting with other people and objects. For realistic replication of such hand motions, high-fidelity hand meshes have to be reconstructed. In this study, we firstly propose DeepHandMesh, a weakly-supervised deep encoder-decoder framework for high-fidelity hand mesh modeling. We design our system to be trained in an end-to-end and weakly-supervised manner; therefore, it does not require groundtruth meshes.
Paper
August 21, 2020 Tiancheng Zhi, Christoph Lassner, Tony Tung, Carsten Stoll, Srinivasa G. Narasimhan, Minh Vo
Paper

TexMesh: Reconstructing Detailed Human Texture and Geometry from RGB-D Video

We present TexMesh, a novel approach to reconstruct detailed human meshes with high-resolution full-body texture from RGBD video. TexMesh enables high quality free-viewpoint rendering of humans. Given the RGB frames, the captured environment map, and the coarse per-frame human mesh from RGB-D tracking, our method reconstructs spatiotemporally consistent and detailed per-frame meshes along with a high-resolution albedo texture.
Paper