Deep Multi-Scale Video Prediction Beyond Mean Square Error

ICLR 2016


Learning to predict future images from a video sequence involves the construction of an internal representation that models the image evolution accurately, and therefore, to some degree, its content and dynamics. This is why pixel-space video prediction may be viewed as a promising avenue for unsupervised feature learning. In addition, while optical flow has been a very studied problem in computer vision for a long time, future frame prediction is rarely approached. Still, many vision applications could benefit from the knowledge of the next frames of videos, that does not require the complexity of tracking every pixel trajectory. In this work, we train a convolutional network to generate future frames given an input sequence. To deal with the inherently blurry predictions obtained from the standard Mean Squared Error (MSE) loss function, we propose three different and complementary feature learning strategies: a multi-scale architecture, an adversarial training method, and an image gradient difference loss function. We compare our predictions to different published results based on recurrent neural networks on the UCF101 dataset.

Related Publications

All Publications

The Web Conference - April 12, 2021

Causal Network Motifs: Identifying Heterogeneous Spillover Effects in A/B Tests

Yuan Yuan, Kristen M. Altenburger, Farshad Kooti

NeurIPS - December 10, 2020

Differentiable Expected Hypervolume Improvement for Parallel Multi-Objective Bayesian Optimization

Samuel Daulton, Maximilian Balandat, Eytan Bakshy

UIST - October 22, 2020

Decoding Surface Touch Typing from Hand-Tracking

Mark Richardson, Matt Durasoff, Robert Wang

IEEE International Symposium on High-Performance Computer Architecture (HPCA) - February 27, 2021

Understanding Training Efficiency of Deep Learning Recommendation Models at Scale

Bilge Acun, Matthew Murphy, Xiaodong Wang, Jade Nie, Carole-Jean Wu, Kim Hazelwood

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy