Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits

Ameet Talwalkar
Giulia DeSalvo
Kevin Jamieson
Lisha Li
5th International Conference on Learning Representations(2017)
Google Scholar


Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian Optimiza- tion to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation. We present H YPERBAND , a novel algorithm for hyperparameter optimization that is simple, flexible, and theoretically sound. H YPERBAND is a principled early-stoppping method that adaptively allocates a pre- defined resource, e.g., iterations, data samples or number of features, to randomly sampled configurations. We compare H YPERBAND with state-of-the-art Bayesian Optimization methods on several hyperparameter optimization problems. We ob- serve that H YPERBAND can provide over an order of magnitude speedups over competitors on a variety of neural network and kernel-based learning problems.

Research Areas