Research Area
Year Published

92 Results

January 1, 2020

Designing Safe Spaces for Virtual Reality

Ethics in Design and Communication

Virtual Reality (VR) designers accept the ethical responsibilities of removing a user’s entire world and superseding it with a fabricated reality. These unique immersive design challenges are intensified when virtual experiences become public and socially-driven. As female VR designers in 2018, we see an opportunity to fold the language of consent into the design practice of virtual reality—as a means to design safe, accessible, virtual spaces.

Publication will be made available in 2020.

By: Michelle Cortese, Andrea Zeller

November 10, 2019

An Integrated 6DoF Video Camera and System Design


Designing a fully integrated 360◦ video camera supporting 6DoF head motion parallax requires overcoming many technical hurdles, including camera placement, optical design, sensor resolution, system calibration, real-time video capture, depth reconstruction, and real-time novel view synthesis. While there is a large body of work describing various system components, such as multi-view depth estimation, our paper is the first to describe a complete, reproducible system that considers the challenges arising when designing, building, and deploying a full end-to-end 6DoF video camera and playback environment.

By: Albert Parra Pozo, Michael Toksvig, Terry Filiba Schrager, Joyce Hsu, Uday Mathur, Alexander Sorkine-Hornung, Richard Szeliski, Brian Cabral

November 9, 2019

Harassment in Social Virtual Reality: Challenges for Platform Governance

Conference on Computer-Supported Cooperative Work and Social Computing (CSCW)

In immersive virtual reality (VR) environments, experiences of harassment can be exacerbated by features such as synchronous voice chat, heightened feelings of presence and embodiment, and avatar movements that can feel like violations of personal space (such as simulated touching or grabbing). Simultaneously, efforts to govern these developing spaces are made more complex by the distributed landscape of virtual reality applications and the dynamic nature of local community norms. To better understand this nascent social and psychological environment, we interviewed VR users (n=25) about their experiences with harassment, abuse, and discomfort in social VR.

By: Lindsay Blackwell, Nicole Ellison, Natasha Elliott-Deflo, Raz Schwartz

September 9, 2019

Flexible binaural resynthesis of room impulse responses for augmented reality research

EAA Spatial Audio Signal Processing Symposium (SASP)

A basic building block of audio for Augmented Reality (AR) is the use of virtual sound sources layered on top of real sources present in an environment. In order to perceive these virtual sources as belonging to the natural scene it is important to carefully replicate the room acoustics of the listening space. However, it is unclear to what extent the real and virtual room impulse responses (RIR) need to be matched in order to generate plausible scenes in which virtual sound sources blend seamlessly with real sound sources. This contribution presents an auralization framework that allows binaural rendering, manipulation and reproduction of room acoustics in augmented reality scenarios, in order to get a better understanding of the perceptual relevance of individual room acoustic parameters.

By: Sebastià V. Amengual Garí, W. Owen Brimijoin, Henrik G. Hassager, Philip W. Robinson
Areas: AR/VR

September 6, 2019

Perceptual comparison of ambisonics-based reverberation methods in binaural listening

EAA Spatial Audio Signal Processing Symposium (SASP)

Reverberation plays a fundamental role in the auralisation of enclosed spaces as it contributes to the realism and immersiveness of virtual 3D sound scenes. However, rigorous simulation of interactive room acoustics is computationally expensive, and it is common practice to use simplified models at the cost of accuracy. In the present study, two subjective listening tests were carried out to explore trade-offs between algorithmic complexity (and approach) and perceived spatialisation quality in a binaural spatialisation context.

By: Isaac Engel, Craig Henry, Sebastià V. Amengual Garí, Philip Robinson, David Poirier-Quinot, Lorenzo Picinali
Areas: AR/VR

September 5, 2019

C3DPO: Canonical 3D Pose Networks for Non-Rigid Structure From Motion

International Conference on Computer Vision (ICCV)

We propose C3DPO, a method for extracting 3D models of deformable objects from 2D keypoint annotations in unconstrained images. We do so by learning a deep network that reconstructs a 3D object from a single view at a time, accounting for partial occlusions, and explicitly factoring the effects of viewpoint changes and object deformations.

By: David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, Andrea Vedaldi

August 12, 2019

Efficient Segmentation: Learning Downsampling Near Semantic Boundaries

International Conference on Computer Vision (ICCV)

Many automated processes such as auto-piloting rely on a good semantic segmentation as a critical component. To speed up performance, it is common to downsample the input frame. However, this comes at the cost of missed small objects and reduced accuracy at semantic boundaries. To address this problem, we propose a new content-adaptive downsampling technique that learns to favor sampling locations near semantic boundaries of target classes.

By: Dmitrii Marin, Zijian He, Peter Vajda, Priyam Chatterjee, Sam Tsai, Fei Yang, Yuri Boykov

July 31, 2019

Neural Volumes: Learning Dynamic Renderable Volumes from Images


To overcome memory limitations of voxel-based representations, we learn a dynamic irregular grid structure implemented with a warp field during ray-marching. This structure greatly improves the apparent resolution and reduces grid-like artifacts and jagged motion. Finally, we demonstrate how to incorporate surface-based representations into our volumetric-learning framework for applications where the highest resolution is required, using facial performance capture as a case in point.

By: Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, Yaser Sheikh

July 12, 2019

The contributions of skin stretch and kinesthetic information to static weight perception

World Haptics

In this study, we examined the contributions of kinesthetic and skin stretch cues, in isolation and together, to the static perception of weight. In two psychophysical experiments, we asked participants either to detect on which hand a weight was presented or to compare between two weight cues. Two closed-loop controlled haptic devices were used to present weights with a precision of 0.05g to an end-effector held in a pinch grasp.

By: Femke E. van Beek, Raymond J. King, Casey Brown, Massimiliano Di Luca
Areas: AR/VR

July 12, 2019

VR Facial Animation via Multiview Image Translation


In this work, we present a bidirectional system that can animate avatar heads of both users’ full likeness using consumer-friendly headset mounted cameras (HMC). There are two main challenges in doing this: unaccommodating camera views and the image-to-avatar domain gap. We address both challenges by leveraging constraints imposed by multiview geometry to establish precise image-to-avatar correspondence, which are then used to learn an end-to-end model for real-time tracking.

By: Shih-En Wei, Jason Saragih, Tomas Simon, Adam W. Harley, Stephen Lombardi, Michal Perdoch, Alexander Hypes, Dawei Wang, Hernan Badino, Yaser Sheikh