Publications

November 9, 2019

Conference on Computer-Supported Cooperative Work and Social Computing (CSCW)

In immersive virtual reality (VR) environments, experiences of harassment can be exacerbated by features such as synchronous voice chat, heightened feelings of presence and embodiment, and avatar movements that can feel like violations of personal space (such as simulated touching or grabbing). Simultaneously, efforts to govern these developing spaces are made more complex by the distributed landscape of virtual reality applications and the dynamic nature of local community norms. To better understand this nascent social and psychological environment, we interviewed VR users (n=25) about their experiences with harassment, abuse, and discomfort in social VR.

Areas: AR/VR Human Computer Interaction & UX

October 29, 2019

International Conference on Computer Vision (ICCV)

We present a 16.2 million frame (50 hour) multimodal dataset of two-person face-to-face spontaneous conversations. Our dataset features synchronized body and finger motion as well as audio data. To the best of our knowledge, it represents the largest motion capture and audio dataset of natural conversations to date.

Areas: AR/VR Computer Vision Machine Learning

October 28, 2019

International Conference on Computer Vision (ICCV)

We present DenseRaC, a novel end-to-end framework for jointly estimating 3D human pose and body shape from a monocular RGB image. Our two-step framework takes the body pixel-to-surface correspondence map (i.e., IUV map) as proxy representation and then performs estimation of parameterized human pose and shape.

Areas: AR/VR Computer Vision Facebook AI Research Machine Learning

October 28, 2019

Journal of Vision

Scientists and engineers have created computations and made measurements that characterize the first steps of seeing. ISETBio software integrates such computations and data into an open-source software package. The initial ISETBio implementations modeled image formation (physiological optics) for planar or distant scenes. The ISET3d software described here extends that implementation, simulating image formation for three-dimensional scenes.

Areas: AR/VR

October 27, 2019

International Conference on Computer Vision (ICCV)

We present Habitat, a platform for research in embodied artificial intelligence (AI). Habitat enables training embodied agents (virtual robots) in highly efficient photorealistic 3D simulation.

Areas: AR/VR Computer Vision Facebook AI Research Machine Learning

September 19, 2019

Symposium on Applied Perception (SAP)

In this study, we analyze the performance, user preference, and sense of ownership for eight virtual grasping visualizations. Six are classified as either a tracked hand visualization or an outer hand visualization. The tracked hand visualizations are those that allow the virtual hand to enter the object being grasped, whereas the outer hand visualizations do not, thereby simulating a realistic interaction.

Areas: AR/VR

September 9, 2019

EAA Spatial Audio Signal Processing Symposium (SASP)

A basic building block of audio for Augmented Reality (AR) is the use of virtual sound sources layered on top of real sources present in an environment. In order to perceive these virtual sources as belonging to the natural scene it is important to carefully replicate the room acoustics of the listening space. However, it is unclear to what extent the real and virtual room impulse responses (RIR) need to be matched in order to generate plausible scenes in which virtual sound sources blend seamlessly with real sound sources. This contribution presents an auralization framework that allows binaural rendering, manipulation and reproduction of room acoustics in augmented reality scenarios, in order to get a better understanding of the perceptual relevance of individual room acoustic parameters.

Areas: AR/VR

September 6, 2019

EAA Spatial Audio Signal Processing Symposium (SASP)

Reverberation plays a fundamental role in the auralisation of enclosed spaces as it contributes to the realism and immersiveness of virtual 3D sound scenes. However, rigorous simulation of interactive room acoustics is computationally expensive, and it is common practice to use simplified models at the cost of accuracy. In the present study, two subjective listening tests were carried out to explore trade-offs between algorithmic complexity (and approach) and perceived spatialisation quality in a binaural spatialisation context.

Areas: AR/VR

September 5, 2019

International Conference on Computer Vision (ICCV)

We propose C3DPO, a method for extracting 3D models of deformable objects from 2D keypoint annotations in unconstrained images. We do so by learning a deep network that reconstructs a 3D object from a single view at a time, accounting for partial occlusions, and explicitly factoring the effects of viewpoint changes and object deformations.

Areas: AR/VR Computer Vision Facebook AI Research

August 12, 2019

International Conference on Computer Vision (ICCV)

Many automated processes such as auto-piloting rely on a good semantic segmentation as a critical component. To speed up performance, it is common to downsample the input frame. However, this comes at the cost of missed small objects and reduced accuracy at semantic boundaries. To address this problem, we propose a new content-adaptive downsampling technique that learns to favor sampling locations near semantic boundaries of target classes.

Areas: AR/VR Computer Vision Facebook AI Research