Peter C. Ma
Research Areas
Authored Publications
Sort By
CATWILD: Compiler Autotuning For TPU workloads in the Wild
Kirill Gugaev
Mike Burrows
Mangpo Phothilimthana
Naveen Kumar
Idan Nachman
Nacho Cano
David Liu
Zongwei Zhou
Tengyu Sun
Berkin Ilbeyi
Jie Sun
Vineetha Govindaraj
Ziqiang Feng
Hyojun Kim
Matheus Camargo
Chao Wang
Alex Wertheim
Domagoj Babic
Aleksey Orekhov
Emma Sevastian
Wenqi Cao
Shobha Vasudevan
Karthik Srinivasa Murthy
Jose Baiocchi Paredes
Emma Wang
Ryan Lefever
Arissa Wongpanich
Νikhil Sarda
Mehrdad Khani
Matt Bierbaum
Sami Abu-El-Haija
Samrat Ghosh
2026
Preview abstract
Compilers play a fundamental role at achieving peak performance for machine learning (ML) workloads. However, given the diverse nature of workloads and accelerators, compilers' heuristics and analytical cost models often result in sub-optimal performance, and thus waste precious datacenter resources. Furthermore, the multitude of tunable parameters and their complex interplay often make it impossible for human experts to manually find optimal configurations. In this paper, we present
CATWILD, a system that automatically optimizes ML jobs in Google's TPU fleet using compiler autotuning techniques. We describe CATWILD’s design and implementation, and evaluate its performance using a handful of representative metrics. We further report experiences and lessons learned from its five-year development and operation. To the best of our knowledge, CATWILD represents the first ML compiler autotuning solution deployed in datacenters at scale. Its successful rollout yielded substantial benefits, optimizing over 70\% of daily TPU training jobs and achieving significant chip savings
View details
Graph Transformer: A Generalized Method for Computation Graph Optimizations
Amirali Abdolrashidi
Anna Darling Goldie
Azalia Mirhoseini
Daniel Wong
Hanxiao Liu
James Pierce Laudon
Mangpo Phothilimthana
Qiumin Xu
Shen Wang
Sudip Roy
Yanqi Zhou
(2020)
Preview abstract
Runtime and scalability of neural networks can be significantly affected by computational graph optimization during compilation. Most existing automated graph optimizations are impractical for deployment due to the significant amount of compute required and their inability to generalize to new, previously held-out graphs. To address both limitations, we propose an end-to-end deep reinforcement learning method named Graph Transformer (GTf), based on a scalable sequential attention mechanism over an inductive graph neural network that is transferable to new, unseen graphs. GTf generates decisions on the entire graph in a single-shot fashion, rather than on each individual node progressively, drastically speeding up the search compared to prior methods. Moreover, we propose recurrent attention layers to jointly optimize dependent graph optimization tasks and demonstrate 33%-60% speedup of three graph optimization tasks compared to Tensorflow default optimizations. On a diverse set of representative graphs consisting of 1k-80k nodes, including Inception-v3, Transformer-XL, and WaveNet, GTf achieves an average 21% improvement over human experts and 18% improvement over the prior art with 15x faster convergence, on a device placement task evaluated in real systems.
View details
