Peter C. Ma
Research Areas
Authored Publications
Sort By
CATWILD: Compiler Autotuning For TPU workloads in the Wild
Ignacio Cano
Yu Emma Wang
Mike Burrows
Ziqiang Feng
Matheus Camargo
Chao Wang
David H. Liu
Tengyu Sun
Alex Wertheim
Arissa Wongpanich
Hyojun Kim
Wenqi Cao
Aleksey Orekhov
Emma Sevastian
Mehrdad Khani
Karthik Srinivasa Murthy
Berkin Ilbeyi
Ryan Lefever
Arjun Khare
Matthew Bierbaum
Jeremiah Wilke
Sami Abu-El-Haija
Νikhil Sarda
Vineetha Govindaraj
Shobha Vasudevan
Kirill Gugaev
Idan Nachman
Jie Sun
Jose Baiocchi Paredes
Samrat Ghosh
Domagoj Babic
Zongwei Zhou
Naveen Kumar
Mangpo Phothilimthana
2026
Preview abstract
Compilers play a fundamental role at achieving peak performance for machine learning (ML) workloads. However, given the diverse nature of workloads and accelerators, compilers' heuristics and analytical cost models often result in sub-optimal performance, and thus waste precious datacenter resources. Furthermore, the multitude of tunable parameters and their complex interplay often make it impossible for human experts to manually find optimal configurations. In this paper, we present
CATWILD, a system that automatically optimizes ML jobs in Google's TPU fleet using compiler autotuning techniques. We describe CATWILD’s design and implementation, and evaluate its performance using a handful of representative metrics. We further report experiences and lessons learned from its five-year development and operation. To the best of our knowledge, CATWILD represents the first ML compiler autotuning solution deployed in datacenters at scale. Its successful rollout yielded substantial benefits, optimizing over 70% of daily TPU training jobs and achieving significant chip savings
View details
Graph Transformer: A Generalized Method for Computation Graph Optimizations
Amirali Abdolrashidi
Anna Darling Goldie
Azalia Mirhoseini
Daniel Wong
Hanxiao Liu
James Pierce Laudon
Mangpo Phothilimthana
Qiumin Xu
Shen Wang
Sudip Roy
Yanqi Zhou
(2020)
Preview abstract
Runtime and scalability of neural networks can be significantly affected by computational graph optimization during compilation. Most existing automated graph optimizations are impractical for deployment due to the significant amount of compute required and their inability to generalize to new, previously held-out graphs. To address both limitations, we propose an end-to-end deep reinforcement learning method named Graph Transformer (GTf), based on a scalable sequential attention mechanism over an inductive graph neural network that is transferable to new, unseen graphs. GTf generates decisions on the entire graph in a single-shot fashion, rather than on each individual node progressively, drastically speeding up the search compared to prior methods. Moreover, we propose recurrent attention layers to jointly optimize dependent graph optimization tasks and demonstrate 33%-60% speedup of three graph optimization tasks compared to Tensorflow default optimizations. On a diverse set of representative graphs consisting of 1k-80k nodes, including Inception-v3, Transformer-XL, and WaveNet, GTf achieves an average 21% improvement over human experts and 18% improvement over the prior art with 15x faster convergence, on a device placement task evaluated in real systems.
View details
