Google Research

State of the Art on Neural Rendering

  • Ayush Tewari
  • Christian Theobalt
  • Dan B Goldman
  • Eli Shechtman
  • Gordon Wetzstein
  • Jason Saragih
  • Jun-Yan Zhu
  • Justus Thies
  • Kalyan Sunkavalli
  • Maneesh Agrawala
  • Matthias Niessner
  • Michael Zollhöfer
  • Ohad Fried
  • Ricardo Martin Brualla
  • Rohit Kumar Pandey
  • Sean Fanello
  • Stephen Lombardi
  • Tomas Simon
  • Vincent Sitzmann
Computer Graphics Forum (2020)

Abstract

The efficient rendering of photo-realistic virtual worlds is a long standing effort of computer graphics. Over the last few years, rapid orthogonal progress in deep generative models has been made by the computer vision and machine learning communities leading to powerful algorithms to synthesize and edit images. Neural rendering approaches are a hybrid of both of these efforts that combine physical knowledge, such as a differentiable renderer, with learned components for controllable image synthesis. Nowadays, neural rendering is employed for solving a steadily growing number of computer graphics and vision problems. This state-of-the-art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photo-realistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. Specifically, we are dealing with the type of control, i.e., how the control is provided, which parts of the pipeline are learned, explicit vs. implicit control, generalization, and stochastic vs. deterministic synthesis. The second half of this state-of-the-art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, re-lighting, free-viewpoint video, and the creation of photo-realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

Research Areas

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work