Pixel Codec Avatars

Conference on Computer Vision and Pattern Recognition (CVPR)


Telecommunication with photorealistic avatars in virtual or augmented reality is a promising path for achieving authentic face-to-face communication in 3D over remote physical distances. In this work, we present the Pixel Codec Avatars (PiCA): a deep generative model of 3D human faces that achieves state of the art reconstruction performance while being computationally efficient and adaptive to the rendering conditions during execution. Our model combines two core ideas: (1) a fully convolutional architecture for decoding spatially varying features, and (2) a rendering-adaptive per-pixel decoder. Both techniques are integrated via a dense surface representation that is learned in a weakly-supervised manner from low-topology mesh tracking over training images. We demonstrate that PiCA improves reconstruction over existing techniques across testing expressions and views on persons of different gender and skin tone. Importantly, we show that the PiCA model is much smaller than the state-of-art baseline model, and makes multi-person telecommunication possible: on a single Oculus Quest 2 mobile VR headset, 5 avatars are rendered in realtime in the same scene.


Related Publications

All Publications

Presentation on Late-Breaking Work at CHI - May 9, 2021

A Multichannel Pneumatic Analog Control System for Haptic Displays

Benjamin Stephens-Fripp, Ali Israr, Carine Rognon

ACM CHI Virtual Conference on Human Factors in Computing Systems (CHI) - May 8, 2021

Armstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual Reality

Zhen Li, Joannes Chan, Joshua Walton, Hrvoje Benko, Daniel Wigdor, Michael Glueck

CHI - May 8, 2021

Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality

Conor R. Foy, John J. Dudley, Aakar Gupta, Hrvoje Benko, Per Ola Kristensson

CHI - May 8, 2021

StickyPie: A Gaze-Based, Scale-Invariant Marking Menu Optimized for AR/VR

Sunggeun Ahn, Stephanie Santosa, Mark Parent, Daniel Wigdor, Tovi Grossman, Marcello Giordano

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy