Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
Abstract
Model-based reinforcement learning agents learn a simulator, a world model of their environment,
to facilitate efficient planning and policy learning. The standard approach is to minimize a predictive
loss, such as Mean Squared Error or KL divergence, which optimizes for one-step, average-case accuracy.
However, reinforcement learning algorithms are powerful optimizers that inevitably discover and exploit
the tiniest inaccuracies in the learned model, leading to simulator exploitation and a reality gap where
policies that succeed in simulation fail catastrophically in the real world. In this paper, we argue that
the objective for learning simulators should not be predictive accuracy, but strategic robustness. We
formulate this as a zero-sum, minimax game between a Model Player (the simulator) and a Policy Player
(the adversary), where the goal is to find a model that is un-exploitable by any policy.
We provide a comprehensive theoretical analysis of this game. First, we provide a regret bound
showing this game is learnable in an online framework. Second, we introduce a general, tractable
simplification of the game using a critic function, and provide finite-time guarantees that bound the true
(global) policy-value gap by the (local) critic’s loss, parameterized by a coverage constant κ. Third,
we introduce our most significant theoretical novelty: the Error-MDP Duality, which proves that the
global problem of finding the worst-case policy is formally dual to a local RL problem of finding an
optimal policy in an Error-MDP where the reward is the one-step critic error. This duality provides
the first principled justification for critic-guided active data selection, which we analyze as an online
convex-concave game and for which we prove a convergence guarantee.
to facilitate efficient planning and policy learning. The standard approach is to minimize a predictive
loss, such as Mean Squared Error or KL divergence, which optimizes for one-step, average-case accuracy.
However, reinforcement learning algorithms are powerful optimizers that inevitably discover and exploit
the tiniest inaccuracies in the learned model, leading to simulator exploitation and a reality gap where
policies that succeed in simulation fail catastrophically in the real world. In this paper, we argue that
the objective for learning simulators should not be predictive accuracy, but strategic robustness. We
formulate this as a zero-sum, minimax game between a Model Player (the simulator) and a Policy Player
(the adversary), where the goal is to find a model that is un-exploitable by any policy.
We provide a comprehensive theoretical analysis of this game. First, we provide a regret bound
showing this game is learnable in an online framework. Second, we introduce a general, tractable
simplification of the game using a critic function, and provide finite-time guarantees that bound the true
(global) policy-value gap by the (local) critic’s loss, parameterized by a coverage constant κ. Third,
we introduce our most significant theoretical novelty: the Error-MDP Duality, which proves that the
global problem of finding the worst-case policy is formally dual to a local RL problem of finding an
optimal policy in an Error-MDP where the reward is the one-step critic error. This duality provides
the first principled justification for critic-guided active data selection, which we analyze as an online
convex-concave game and for which we prove a convergence guarantee.
