Craig Boutilier

Craig Boutilier

Craig Boutilier is Principal Scientist at Google. He works on various aspects of decision making under uncertainty, with a current focus on sequential decision models: reinforcement learning, Markov decision processes, temporal models, etc.

Positions and Appointments:
He was a Professor in the Department of Computer Science at the University of Toronto (on leave) and Canada Research Chair in Adaptive Decision Making for Intelligent Systems. He received his Ph.D. in Computer Science from the University of Toronto in 1992, and worked as an Assistant and Associate Professor at the University of British Columbia from 1991 until his return to Toronto in 1999. He served as Chair of the Department of Computer Science at Toronto from 2004-2010. He was co-founder (with Tyler Lu) of Granata Decision Systems from 2012-2015, until his move to Google in 2015.

Boutilier was a consulting professor at Stanford University from 1998-2000, an adjunct professor at the University of British Columbia from 1999-2010, and a visiting professor at Brown University in 1998, at the University of Toronto in 1997-98, at Carnegie Mellon University in 2008-09, and at Université Paris-Dauphine (Paris IX) in the spring of 2011. He served on the Technical Advisory Board of CombineNet, Inc. from 2001 to 2010.

Research:
Boutilier's current research efforts focus on various aspects of decision making under uncertainty, including the use of generative models and LLMs, in areas such as: recommender systems, preference modeling and elicitation, mechanism design, game theory and multiagent decision processes, economic models, social choice, computational advertising, Markov decision processes, reinforcement learning and probabilistic inference. His research interests have spanned a wide range of topics, from knowledge representation, belief revision, default reasoning, and philosophical logic, to probabilistic reasoning, decision making under uncertainty, multiagent systems, and machine learning.

Research & Academic Service:
Boutilier is a past Editor-in-Chief of the Journal of Artificial Intelligence Research (JAIR). He was a past Associate Editor with the ACM Transactions on Economics and Computation (TEAC), the Journal of Artificial Intelligence Research (JAIR), the Journal of Machine Learning Research (JMLR), and Autonomous Agents and Multiagent Systems (AAMAS); and he has sat on the editorial/advisory boards of several other journals. Boutilier has organized several international conferences and workshops, including his work as Program Chair of the Twenty-first International Joint Conference on Artificial Intelligence (IJCAI-09) and Program Chair of the Sixteenth Conference on Uncertainty in Artificial Intelligence (UAI-2000). He has also served on the conference program committees of roughly 75 leading international conferences.

He will serve as Conference Chair of the Thirty-seventh International Joint Conference on Artificial Intelligence (IJCAI-28).

Awards and Honors:
Boutilier is a Fellow of the Royal Society of Canada (RSC), the Association for Computing Machinery (ACM) and the Association for the Advancement of Artificial Intelligence (AAAI). He was the recipient of the 2018 ACM/SIGAI Autonomous Agents Research Award, He was awarded a Tier I Canada Research Chair, an Isaac Walton Killam Research Fellowship, and an IBM Faculty Award. He received the Killam Teaching Award from the University of British Columbia in 1997. He has also received a number of Best Paper awards including:

Authored Publications
Sort By
  • Title
  • Title, descending
  • Year
  • Year, descending
Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action-Spaces
Haitong Ma
Ofir Nabati
Bo Dai
Na Li
Shie Mannor
Guy Tennenholtz
Proceedings of the 43rd International Conference on Machine Learning (ICML-26), Seoul, South Korea (2026)
Preview abstract Reinforcement learning (RL) algorithms have achieved superhuman performance on many sequential decision-making tasks, but often struggle in domains with large, combinatorial action spaces. To address this, we introduce a practical and stable algorithm for training discrete diffusion models to represent policies in such environments. We formulate a policy mirror descent algorithm that enhances training stability by reframing policy optimization as an inference problem, which naturally aligns with the learning objective of discrete diffusion models. Through extensive experiments on a suite of challenging benchmark tasks, we demonstrate that our approach achieves significant improvements over existing methods in both performance and sample efficiency. This work opens a promising new direction for applying discrete diffusion models in RL to tackle long-standing challenges in large-scale combinatorial action spaces. View details
MaxIM: Maximally Informative Incremental Summarization via Reinforcement Learning
Jihwan Jeong
Guy Tennenholtz
Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS-26), Sydney (2026)
Preview abstract Continuous streaming applications require AI systems to maintain bounded episodic memories of unbounded interaction histories. While optimizing these memory updates for downstream task utility is the ultimate goal, standard reinforcement learning (RL) suffers from severe credit assignment bottlenecks due to sparse, delayed rewards. We introduce MaxIM (Maximally Informative Language Model), a framework that formulates incremental summarization as a capacity-constrained, information-theoretic reward shaping problem. Grounded in value suboptimality bounds, we mathematically prove that minimizing sequential information loss requires maximizing the predictive sufficiency of the summary. To prevent LLMs from hallucinating during this predictive optimization, we introduce a secondary recursive consistency constraint that enforces strict historical grounding. Because computing true mutual information is intractable, MaxIM leverages oracle context during offline training to learn variational lower bounds via a parameterized critic. This provides the RL policy with dense, step-by-step regularizers that solve the sparse credit assignment problem. Empirical evaluation on long-horizon benchmarks---including multi-turn user simulation and goal-directed scientific reading---demonstrates that MaxIM achieves significant improvements across comprehensive autorater metrics and downstream task utility compared to state-of-the-art baselines, all without requiring expensive human preference labels. View details
Diffusion Controller: Framework, Algorithms and Parameterization
Tong Yang
Moonkyung Ryu
Guy Tennenholtz
Yuejie Chi
Bo Dai
Proceedings of the 43rd International Conference on Machine Learning (ICML-26), Seoul, South Korea (2026)
Preview abstract Controllable generation with diffusion models is often treated as a collection of heuristics rather than a unified optimization problem. We propose a principled control formulation by viewing the diffusion reverse process as an instance of a (generalized) linearly-solvable Markov decision process (LS-MDP). This perspective turns controllable generation into regularized optimal control around a pretrained diffusion policy, yielding tractable objectives and algorithmic updates. Under this framework, we study two practical finetuning regimes. When paired target data are available, we obtain a supervised finetuning (SFT) objective. When only a terminal reward model is available, we derive reinforcement-learning finetuning (RLFT) methods from the LS-MDP solution structure, including (i) a reward-weighted regression loss and (ii) a policy-gradient approach (with standard extensions such as PPO). Crucially, the LS-MDP optimality conditions imply an explicit relationship between the optimal and pretrained score functions. We leverage this to derive a new score-function parameterization that isolates the control signal and enables “gray-box” finetuning with substantially fewer trainable parameters. Experiments across SFT and RLFT show this parameterization improves over existing finetuning baselines while achieving stronger sample/parameter efficiency. View details
Controllable User Simulation
Guy Tennenholtz
Ofer Meshi
Uri Shalit
Jihwan Jeong
Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS-26), Sydney (2026)
Preview abstract Evaluating conversational agents with static offline datasets, while reliable, can be rigid, failing to cover rare scenarios or support testing new policies. This has induced increased reliance on user simulators for targeted, counterfactual evaluation. However, the theoretical foundations of these simulators remain understudied. In this work, we formalize controllable simulation as generation conditional on a random variable representing observable or latent targets of interest. By bridging natural language evaluation with classical off-policy evaluation (OPE) mathematics, we show that if controls are determined by full conversation trajectories, they couple to the data-generating agent policy, introducing a lookahead bias that breaks causal consistency. We also prove that under covariate shift, the variance of the trajectory density ratio explodes geometrically---a structural manifestation of the curse of horizon, which we term controllability collapse. We demonstrate mitigation strategies including strictly a priori controls, step-wise dynamic controls, and direct policy-conditioned learning. Empirical evaluation on WildChat and ConvApparel benchmarks show that static global controls distort conversational distributions and collapse behavioral diversity, while inference-time rejection sampling is computationally intractable. Conversely, our causally grounded simulators preserve natural variance and exhibit robust zero-shot generalization to unseen agent behaviors. View details
Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion
Pengcheng Jiang
Judith Yue Li
Moonkyung Ryu
R. Lily Hu
Kun Su
Liam Hebert
Hao Peng
Jiawei Han
Dima Kuzmin
2026
Preview abstract Many modern retrieval problems are set-valued: given a broad intent, the system must return a collection of results that optimizes higher-order properties (e.g., diversity, coverage, complementarity, coherence) while staying grounded to a fixed database. Set-valued objectives are inherently non-decomposable and are not captured by existing supervised (query, content) datasets which only prioritize top-1 retrieval. While reinforcement learning (RL) can optimize set-level objectives via interaction, deploying an RL-tuned LLM for fan-out retrieval is prohibitively expensive at query time. Conversely, diffusion-based generative retrieval enables efficient single-pass fan-out in embedding space, but requires objective-aligned training targets. To address these issues, we propose R4T (Retrieve-for-Train), which uses RL once as an objective transducer in a three step process: (i) train a fan-out LLM with composite set-level rewards, (ii) synthesize objective-consistent training pairs, and (iii) train a lightweight diffusion retriever to model the conditional distribution of set-valued outputs. Across Polyvore and a music playlist dataset, R4T improves retrieval quality over strong baselines while reducing query-time fan-out latency by an order of magnitude. View details
Preview abstract While modern text-to-image (T2I) models excel at generating high-quality images, they are typically trained to optimize with respect to generalized, population-level preferences. This homogeneous approach ignores the diverse, individual tastes and aesthetic judgments of different users. In this work, we propose a novel framework that learns fine-grained user preferences without relying on computationally expensive visual language models (VLMs) or prompt-sensitive text profiles. Instead, we introduce a robust, continuous user representation that models a user’s reward function as a linear combination of K base user types. We learn user-specific weights $\lambda_u$ via logistic regression on pairwise preference data to construct a continuous user embedding. This embedding is integrated into the diffusion process via an IP-Adapter, and fine-tuned using Diffusion-DPO. Our approach consistently generates images aligned with individual reward functions, achieving a 66.2% win rate against a pre-trained SDXL baseline and a 63.2% win rate against the state-of-the-art PPD framework. View details
ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders
Ofer Meshi
Guy Tennenholtz
Jihwan Jeong
Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL-26), Rabat, Morocco (2026), pp. 5270-5304
Preview abstract LLM-based user simulators are a scalable solution for improving conversational AI, but a critical realism gap undermines their effectiveness. To close this gap, we introduce a framework for building and validating high-fidelity simulators. We present a novel dataset of human-AI shopping conversations designed to capture a wide spectrum of user experiences. To measure fidelity, we propose a hybrid evaluation protocol that combines statistical alignment with a learned, discriminator-based Human-Likeness Score. Our most sophisticated simulator, trained via reinforcement learning with iterative critique, achieves a significant leap in realism. Critically, we demonstrate through counterfactual validation that our simulator—trained exclusively on optimal interactions—realistically adapts its behavior to suboptimal system responses, mirroring real user reactions and marking a key advance in creating reliable simulators for robust AI development. View details
Efficient, Property-Aligned Fan-Out Retrieval via RL-Compiled Diffusion
Patrick Jiang
Judith Li
Moonkyung Ryu
Lily Hu
Kun Su
Liam Hebert
Hao Peng
Jiawei Han
Dima Kuzmin
Proceedings of the 43rd International Conference on Machine Learning (ICML-26), Seoul, South Korea (2026)
Preview abstract Many modern retrieval problems are set-valued: given a broad intent, the system must return a collection of results that optimizes higher-order properties (e.g., diversity, coverage, complementarity, coherence) while staying grounded to a fixed database. These objectives are inherently non-decomposable, creating a training bottleneck because property-aligned (query, content) supervision is scarce. Reinforcement learning (RL) can optimize set-level objectives via interaction, but deploying an RL-tuned LLM for fan-out retrieval is expensive at query time. Diffusion-based generative retrieval enables efficient single-pass fan-out in embedding space, but requires objective-aligned training targets. We propose R4T (Retrieve-for-Train), which uses RL once as an objective transducer: (i) train a fan-out LLM with composite set-level rewards, (ii) synthesize objective-consistent training pairs, and (iii) train a lightweight diffusion retriever to model the conditional distribution of set-valued outputs. Across Polyvore and a large-scale music playlist dataset, R4T improves retrieval quality over strong baselines while reducing query-time fan-out latency by an order of magnitude. View details
Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)
Moonkyung Ryu
Mohammad Ghavamzadeh
GENNEXT@SIGIR’25: The 1st Workshop on Next Generation of IR and Recommender Systems with Language Agents, Generative Models, and Conversational AI (2025)
Preview abstract Large language models (LLM), despite their success in conducting natural conversations and solving various challenging NLP tasks, may not aim to understand and solicit a user’s preferences and suggest recommendations using the learned user preferences through the interactions. One primary challenge for pursuing this task is the lack of rich conversational recommendation data sets (of user/agent dialog conversations) that contain diverse preference elicitation scenarios on top of item recommendations. While several standard conversational recommender datasets do contain dialogs that the agents query users for preferences, they are often restricted to the use of particular key-phrases that limit users’ preference expressions w.r.t. particular item features (e.g., location, price, rating, etc). On the other hand, hiring human raters to create a personalized conversational recommender dataset that consists of rich preference queries (with the use of various abstract preference-describing attributes) and recommendations is a very expensive, error-prone, and time-consuming process because it requires data collection on numerous personalized recommendation scenarios, where each of them may involve exponential possibilities of preference elicitation interactions (with many different queries). We propose a synthetic data generation methodology that utilizes a synthetic recommendation dialog simulator that generates templatized dialogs and uses Gemini Ultra LLM to in-paint the templatized dialogs so that the dialogs become linguistically more natural. The simulator generates recommendation dialogs that align to the user preferences that are represented by embeddings. View details
Preference Adaptive and Sequential Text-to-Image Generation
Ofir Nabati
Guy Tennenholtz
Moonkyung Ryu
Sean Li
42nd International Conference on Machine Learning (ICML-25), Vancouver (2025), pp. 45362-45394
Preview abstract We consider the problem of sequential text-to-image generation. Specifically, we formulate a personalized interactive framework, where an agent iteratively improves a user's prompt through a series of sequential prompt expansions. We formulate the problem as a sequential decision-making task. Using human raters, we create a dataset of sequential preferences for this problem. We then leverage our sequential data, together with large-scale open-source non-sequential datasets to construct user-preference and user-choice models. Particularly, we employ an EM strategy to develop a personalized sequential user model. We then leverage a multi-modal large language model (MM-LLM) and a value-based reinforcement learning (RL) agent to suggest a personalized and diverse slate of prompt expansions to the user. Our Personalized And Sequential Text-to-image Agent (PASTA) empowers diffusion models with personalized multi-turn capabilities, fostering collaborative co-creation, and addressing uncertainties or under-specifications in user intent. We evaluate our agent using human raters, showing significant improvement compared to baseline methods. We also release our sequential rater dataset and additional simulated data of user-agent interactions to advance future research in personalized multi-turn text-to-image generation. View details
×