Blake Richards
Blake Richards is Research Scientist with the Paradigms of Intelligence team at Google. He also works part-time as an Associate Professor at McGill University and is a Core Faculty Member at Mila (the Quebec AI Instititute).
Research Areas
Authored Publications
Sort By
Looking to the brain to improve energy efficiency of AI
Taro Toyoizumi
Hakwan Lau
Michał Klincewicz
Seng Bum Michael Yoo
Megan Peters
taylor.w.webb@gmail.com
Current Biology (2026)
Preview abstract
Modern artificial intelligence (AI) systems have achieved remarkable capabilities, but at an extraordinary energy cost. Training and running large-scale models can consume vast resources, posing environmental, economic, and social challenges. In contrast, biological brains perform lifelong learning, adaptive control, and flexible reasoning using orders of magnitude less energy for learning and adaptation over a lifetime. What accounts for this difference -- and how can it guide future AI development? In this article, we identify key biological principles that support energy-efficient capacities in biological brains, and consider how they might inform the design of more sustainable artificial systems. We organize our analysis around three domains: architectural constraints, signaling strategies, and learning algorithms. In each domain, we discuss concrete observations from biology -- from cell to circuit to cognitive level -- and describe how current and emerging AI systems mirror or diverge from these motifs. One striking feature of biological energy optimization is often overlooked: that brains are remarkably stable in their energy usage across heterogeneous modes, suggesting they may minimize energy needs during active environmental processing through maximizing the utility of “rest-like” background processes. Overall, rather than advocating for biomimicry for its own sake, we argue for biologically informed engineering. Understanding how natural systems minimize energetic cost while maximizing flexibility may help us build AI that is not only powerful, but also efficient, equitable, and environmentally responsible.
View details
Tracing the Representation Geometry of Language Models from Pretraining to Post-training
Guillaume Lajoie
Arna Ghosh
Kumar Krishna Agrawal
Komal Kumar Teru
Melody Zixuan Li
Adam Santoro
2025
Preview abstract
Complex representational changes in Large Language Models (LLMs) are critical for their capabilities, but are often obscured by standard metrics used to evaluate models during training, like loss or gradient norms. Here, we examine the representational changes that occur during LLM pretraining by analyzing their high-dimensional representation geometry using spectral methods (αReQ, RankMe). In two different families of models (OLMo and Pythia), hidden beneath the near monotonically-decreasing loss and gradient norm, we uncover non-monotonic learning phases in the geometry of the representations. These phases are curves. Specifically, we find that the pretraining stage consistently exhibits three distinct phases: (1) a ‘warm-up’ phase where the dimensionality of the representations drops drastically, (2) an ’entropy-seeking’ phase that expands the effective dimensionality of the representations in all directions, and (3) a ’compression-seeking’ phase that reduces the dimensionality by selectively expanding only along the dominant representational axes. This evolving representation geometry governs the trade-off between fitting the training distribution and generalizing beyond it: The models get better at reproducing specific short-context sequences from the data during the entropy-seeking phase, and at generalizing to novel long-context dependencies during the compression-seeking phase. Continued pretraining can lead to additional entropy-seeking and compression-seeking phases. Crucially, we also find that these different phases have implications for downstream fine-tuning. Optimal adaptability for Supervised Fine-Tuning (SFT) emerges significantly earlier than peak zero-shot performance on factual question answering tasks and aligns with the transition out of the first compression-seeking phase. Furthermore, we observe that SFT often induces an ’entropy-seeking’ dynamic whereas Reinforcement Learning from Verifiable Rewards (RLVR) induces a ’compression-seeking’ dynamic. We investigate the implications of these representational dynamics on downstream generalization of instruction-tuning, and exploration capabilities of RLVR-tuned models. Our results demonstrate that spectral methods for analyzing high-dimensional representations can provide new insights on the functionally relevant changes that occur in LLMs over pretraining.
View details
Preview abstract
Biological and artificial intelligence systems navigate the fundamental efficiency-robustness tradeoff for optimal encoding, i.e., they must efficiently encode numerous attributes of the input space while also being robust to noise. This challenge is particularly evident in hierarchical processing systems like the human brain. With a view towards understanding how systems navigate the efficiency-robustness tradeoff, we turned to a population geometry framework for analyzing representations in the human visual cortex alongside artificial neural networks (ANNs). In the ventral visual stream, we found general-purpose, scale-free representations characterized by a power law-decaying eigenspectrum in most but not areas. Of note, certain higher-order visual areas did not have scale-free representations, indicating that scale-free geometry is not a universal property of the brain. In parallel, ANNs trained with a self-supervised learning objective also exhibited scale-free geometry, but not after fine-tuning on a specific task. Based on these empirical results and our analytical insights, we posit that a system’s representation geometry is not a universal property and instead depends upon the computational objective.
View details
