Tomas Pfister
Tomas Pfister is the Head of Cloud AI Research. He came to Google from Apple where he cofounded Apple's central AI research group and published Apple’s first research paper that won the Best Paper Award at CVPR’17. Tomas’ key scientific achievements have been proposing a method to improve the realism of synthetic images; developing the first automated method to detect facial micro-expressions; and inventing a new way for neural networks to exploit spatiotemporal structure. He is currently exploring learning from small amount of labeled data (using techniques such as generative models, few-shot learning, transfer learning) and explainability/interpretability of deep learning models, and is particularly excited about the potential of AI in healthcare & education. His research has laid the foundation for several applications such as Face ID in iPhone X, autonomous driving, human pose estimation, detecting facial micro-expressions & translating sign language. Tomas did his PhD in deep learning with Prof Andrew Zisserman at Oxford University and bachelor’s degree in computer science at Cambridge University. He is the recipient of the Forbes 30 Under 30 award, and has received over 40 research awards, including 3 best paper awards, with numerous publications in top AI research venues. His work has been frequently featured in mainstream media, including Forbes, BusinessInsider & Wired.
Authored Publications
Sort By
Preview abstract
Automating AI research differs from general software engineering due to computationally expensive evaluation (e.g., model training) and opaque performance attribution. Current LLM-based agents struggle here, often generating monolithic scripts that ignore execution costs and causal factors. We introduce MARS (Modular Agent with Reflective Search), a framework optimized for autonomous AI research. MARS relies on three pillars: (1) Budget-Aware Planning via cost-constrained Monte Carlo Tree Search (MCTS) to explicitly balance performance with execution expense; (2) Modular Construction, employing a "Design-Decompose-Implement" pipeline to manage complex research repositories; and (3) Comparative Reflective Memory, which addresses credit assignment by analyzing solution differences to distill high-signal insights. MARS achieves state-of-the-art performance among open-source frameworks on MLE-Bench under comparable settings, maintaining competitiveness with the global leaderboard's top methods. Furthermore, the system exhibits qualitative "Aha!" moments, where 63% of all utilized lessons originate from cross-branch transfer, demonstrating that the agent effectively generalizes insights across search paths.
View details
Preview abstract
The exponential growth of machine learning submissions has strained the traditional peer review process, resulting in slow feedback loops for authors and an immense burden on reviewers to rigorously audit technical soundness and verify literature. To address this, we introduce ScholarPeer, a multi-agent framework designed to operationalize the rigorous auditing workflow of a senior researcher. Rather than attempting to replace human judgment, ScholarPeer serves as a co-scientist: acting as a mentor for rapid author iteration prior to submission, and as an active verification assistant that augments human reviewers. The framework structurally decouples contextualization from critique by deploying a sub-domain historian to synthesize the field's trajectory, a baseline scout to proactively hunt for omitted state-of-the-art comparisons, and a multi-aspect Q&A engine that deeply audits technical soundness-scrutinizing internal logical consistency, experimental validity, and mathematical rigor-while cross-referencing claims against top-tier academic venues. We comprehensively evaluate ScholarPeer on ~1,800 ICLR submissions spanning 2020 through 2025. Our results show that ScholarPeer achieves significant win-rates against state-of-the-art fine-tuned models and search-augmented agentic baselines.
View details
Preview abstract
Understanding long visual documents, where information is distributed across extensive pages of text and graphics, remains a critical challenge for modern Vision-Language Models (VLMs). This difficulty is rooted in two fundamental obstacles: poor evidence localization and a high tendency for model hallucination. To address these issues, we propose DocLens, a multi-agent framework that decomposes the task into two specialized stages.
First, a Lens Module leverages document parsing tools for fine-grained, hierarchical evidence localization at both the page and element level. Second, a Reasoning Module employs a sampling-adjudication mechanism to systematically analyze the localized evidence, mitigating hallucination and synthesizing reliable answers.
Paired with Gemini-2.5-Pro, DocLens achieves state-of-the-art performance on MMLongBench-Doc and FinRAGBench-V, even surpassing human experts. Furthermore, our framework offers a highly cost-effective variant that delivers comparable performance to strong baselines at a five-fold reduction in cost.
View details
PaperBanana: Automating Academic Illustration for AI Scientists
Dawei Zhu
Sujian Li
International Conference on Machine Learning (2026)
Preview abstract
While AI scientists increasingly automate research tasks through advanced language models, generating publication-ready illustrations remains a labor-intensive bottleneck in the scientific workflow. To lift this burden, we introduce PaperBanana, an agentic framework for automated generation of publication-ready academic diagrams. Powered by Nano-Banana-Pro and Gemini-3-Pro, PaperBanana orchestrates a team of specialized agents to retrieve reference examples, devise detailed plans for content and style, render the image, and perform iterative refinement based on self-critique. To rigorously evaluate our framework and address the absence of dedicated benchmarks for automated academic illustration, we introduce PaperBananaBench, comprising 292 test cases for methodology diagrams curated from NeurIPS 2025 publications.
Comprehensive experiments demonstrate that PaperBanana consistently outperforms vanilla Nano-Banana-Pro across all four dimensions—faithfulness, conciseness, readability, and aesthetics—achieving human-level performance. We further show that PaperBanana seamlessly extends to statistical plots through targeted adaptations. Collectively, PaperBanana enables AI scientists to fully automate the generation of publication-ready academic illustrations.
View details
TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems
MD ATIK AHAMED
Qiang Cheng
Forty-third International Conference on Machine Learning (2026)
Preview abstract
Time-series forecasting has traditionally been evaluated solely on numerical accuracy, treating models as "black boxes'' that fail to capture the underlying reasoning. To address this gap, we introduce TFRBench, a novel benchmark designed to evaluate the reasoning capabilities of forecasting systems alongside their numerical accuracy. Unlike existing benchmarks, TFRBench requires models to generate verifiable natural language reasoning by analyzing cross-channel dependencies, identifying strategic trends, and justifying significant events using external context. To construct this benchmark, we propose a systematic multi-agent framework comprising Reasoning, Search, Verifier, Forecasting, and Summary agents. Our benchmark spans five diverse domains including Energy, Sales, Web/CloudOps, Transportation, and Finance, covering 10 distinct datasets. Qualitative evaluation confirms that our generated reasoning is highly faithful and effective; specifically, Large Language Models (LLMs) prompted with our generated reasoning demonstrate significantly improved forecasting accuracy compared to direct forecasting with LLMs. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (shows lower LLM-as-Judge scores) and direct numerical forecasting (MAE and MASE), frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting.
View details
Co-RedTeam: Orchestrated Security Discovery and Exploitation with LLM Agents
Pengfei He
Ash Fox
Stefan Friedli
Daniel Fabian
Burak Gokturk
Jiliang Tang
ICML 2026
Preview abstract
Large language models (LLMs) have shown promise in assisting cybersecurity tasks, yet existing approaches struggle with automatic vulnerability discovery and exploitation due to limited interaction, weak execution grounding, and a lack of experience reuse. We propose Code-RedTeam, a security-aware multi-agent framework designed to mirror real-world red-teaming workflows by integrating security-domain knowledge, code-aware analysis, execution-grounded iterative reasoning, and long-term memory. Code-RedTeam decomposes vulnerability analysis into coordinated discovery and exploitation stages, enabling agents to plan, execute, validate, and refine actions based on real execution feedback while learning from prior trajectories. Extensive evaluations on challenging security benchmarks demonstrate that Code-RedTeam consistently outperforms strong baselines across diverse backbone models, achieving over 60% attack success rate in vulnerability exploitation and up to 10% absolute improvement in vulnerability detection. Ablation and iteration studies further confirm the critical role of execution feedback, structured interaction, and memory for building robust and generalizable cybersecurity agents.
View details
Causal Armor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution
Minbeom Kim
Phil Wallis
Kyomin Jung
Dj Dvijotham
2026
Preview abstract
AI agents equipped with tool-calling capabilities are susceptible to \emph{Indirect Prompt Injection} (IPI) attacks. In this attack scenario, malicious commands hidden within \emph{untrusted} content trick the agent into performing unauthorized actions. Existing defenses can reduce attack success but often suffer from the \emph{over-defense dilemma}: they deploy expensive, \emph{always-on} sanitization that degrades utility and latency even in benign scenarios. We revisit IPI through an operational causal lens: a successful injection manifests as a \emph{grounding collapse} where the user request no longer provides decisive support for the agent's privileged action, while a particular untrusted segment provides disproportionate marginal support. Based on this signature, we propose \texttt{CausalArmor}, a selective defense framework that (i) computes lightweight, normalized leave-one-out attributions at privileged decision points, and (ii) triggers targeted sanitization only when an untrusted segment dominates the user intent. Additionally, CausalArmor employs \emph{retroactive Chain-of-Thought masking} to prevent the agent from acting on ``poisoned" reasoning traces. Experiments on AgentDojo and DoomArena demonstrate that CausalArmor matches the security of aggressive defenses with explainability while preserving utility and latency of AI agents.
View details
VISTA: A Test-Time Self-Improving Video Generation Agent
Xuan Long Do
Hootan Nakhost
The IEEE/CVF Conference on Computer Vision and Pattern Recognition (to appear) (2026)
Preview abstract
Despite rapid advances in text-to-video (T2V) synthesis, generated video quality remains critically dependent on precise user prompts. Existing test-time optimization methods, successful in other domains, struggle with the multi-faceted nature of video. To address this, we introduce VISTA, a novel multi-agent system that autonomously refines prompts to improve video generation. VISTA operates in an iterative loop, first decomposing a user's idea into a structured temporal plan. After generation, the best video is identified through a robust pairwise tournament. This winning video is then critiqued by a trio of specialized agents focusing on visual, audio, and contextual fidelity. Finally, a reasoning agent synthesizes this feedback to introspectively rewrite and enhance the prompt for the next generation cycle. To rigorously evaluate our proposed approach, we introduce MovieGen-Bench, a new benchmark of diverse single- and multi-scene video generation tasks. Experiments show that while prior methods yield inconsistent gains, VISTA consistently improves video quality, achieving up to 60% pairwise win rate against state-of-the-art baselines. Human evaluators concur, preferring VISTA's outputs in 68% of comparisons.
View details
Budget-Aware Tool Use Enables Effective Agent Scaling
Tengxiao Liu
Jin Miao
I-Hung Hsu
Samira Daruki
COLM 2026 (2026)
Preview abstract
Scaling test-time computation improves performance across different tasks on large language models (LLMs), yet mainstream scaling methods remain challenging for tool-augmented LLM agents. Sequential scaling tends to yield shallow tool use and under-exploration, whereas parallel scaling inflates cost through repeated tool calls. The dual costs of tokens and tool calls further complicate cost accounting and hinder fair comparison. In this work, we study the test-time scaling of widely used, tool-reliant search agents under resource constraints, analyzing performance with a unified cost metric that incorporates both tokens and tool calls. To this end, we propose Cost-effective Agent Test-Time Scaling (CATS), a budget-aware framework designed to support more cost-effective scaling by guiding resource allocation between sequential and parallel exploration. Experiments across search-intensive benchmarks show that CATS produces more favorable scaling curves, attaining higher accuracy with fewer tool calls and lower overall cost. Our work introduces a cost-conscious design for agent test-time scaling and contributes empirical insights that enable a more transparent and principled understanding of scaling in tool-augmented agents.
View details
VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation
Dj Dvijotham
Mirko Montanari
ACL 2026
Preview abstract
Artificial intelligence is rapidly evolving, marked by the emergence of Large Language Model (LLM) agents – systems capable of complex reasoning, planning, and interaction with digital and physical environments. These agents, powered by advancements in LLMs, demonstrate remarkable capabilities across diverse domains, including finance, healthcare, web navigation, software development, and daily task assistance. Unlike traditional AI systems, LLM agents can perceive their surroundings, formulate multi-step plans, utilize external tools and APIs, access memory or knowledge bases, and execute actions to achieve specified goals. This ability to act upon the world, however, introduces significant safety and security challenges.
The safety paradigms developed for traditional LLMs, primarily focused on mitigating harmful textual outputs (e.g., toxicity, bias), are insufficient for safeguarding LLM agents. Agents interacting with dynamic environments and executing actions present a broader attack surface and new categories of risk. These include performing unsafe operations, violating privacy constraints through improper data handling or access control failures, deviating from user objectives (task misalignment), and susceptibility to novel manipulation techniques like indirect prompt injection and memory poisoning. Ensuring the trustworthy operation of these powerful agents is paramount, especially as they are integrated into high-stakes applications. To address this critical challenge, we introduce VeriGuard, a novel framework designed to enhance the safety and reliability of LLM agents by interactively verifying their policies and the actions. VeriGuard integrates a verification module that intercepts code-based actions proposed by the agent. In the first step, VeriGuard will generates and verifies the policies. The policies are rigorously checked against a set of predefined safety and security specifications Then each action will be verified to make sure it will align with the agent specification. This interactive verification loop ensures that the agent's behavior remains within safe operational bounds, effectively preventing the execution of harmful or unintended operations. By verifying each step, VeriGuard provides a robust safeguard, substantially improving the trustworthiness of LLM agents in complex, real-world environments.
View details
