Google Research

Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning

  • Yiding Jiang
  • Parth Natekar
  • Manik Sharma
  • Sumukh K. Aithal
  • Dhruva Kashyap
  • Natarajan Subramanyam
  • Carlos Lassance
  • Daniel M. Roy
  • Gintare Karolina Dziugaite
  • Suriya Gunasekar
  • Isabelle Guyon
  • Pierre Foret
  • Scott Yak i
  • Hossein Mobahi
  • Behnam Neyshabur
  • Samy Bengio
Proceedings of the NeurIPS 2020 Competition and Demonstration Track, PMLR (2021)


Deep learning has been recently successfully applied to an ever larger number of problems, ranging from pattern recognition to complex decision making. However, several concerns have been raised, including guarantees of good generalization, which is of foremost importance. Despite numerous attempts, conventional statistical learning approaches fall short of providing a satisfactory explanation on why deep learning works. In a competition hosted at the Thirty-Fourth Conference on Neural Information Processing Systems (NeurIPS 2020), we invited the community to design robust and general complexity measures that can accurately predict the generalization of models. In this paper, we describe the competition design, the protocols, and the solutions of the top-three teams at the competition in details. In addition, we discuss the outcomes, common failure modes, and potential future directions for the competition.

Research Areas

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work