Google Research

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

  • Danyang Zhuo
  • Hao Zhang
  • Ion Stoica
  • Joseph E. Gonzalez
  • Lianmin Zheng
  • Yanping Huang
  • Yida Wang
  • Yonghao Zhuang
  • Yuanzhong Xu
  • Zhifeng Chen
  • Zhuohan Li
16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), USENIX Association (2022), pp. 559-578

Abstract

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations. They do not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive efficient parallel execution plans at each parallelism level. Alpa implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans. Alpa's source code is publicly available at https://github.com/alpa-projects/alpa

Research Areas

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work