Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster and, Better

Deep Learning has revolutionized the fields of Computer Vision, Natural Language, Speech, Information Retrieval and more. However, with the growth of Deep Learning models, the number of parameters, latency, resources required to train, all have increased significantly. Consequently, it has become important to focus on the footprint of the model, not just its quality. We present and motivate the problem of efficiency in Deep Learning, followed by a thorough survey of the five core areas of model efficiency and the seminal work there. We also present an experiment-based guide for practitioners to optimize their models. We believe this is the first comprehensive survey in the Efficient Deep Learning space. Our hope is that this survey would provide the reader with both the mental model and the necessary understanding of the field to firstly apply generic efficiency techniques to immediately get a sizeable improvements, and secondly ideas for experimentation to achieve additional gains.

• Machine Intelligence