Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Exploration

To address the exploration bottleneck in reinforcement learning—particularly in sparse-reward environments where data is scarce—a new paradigm is proposed that explicitly decouples exploration from exploitation. Rather than relying on the computationally expensive dominant approach of training agents with intrinsic motivation via policy optimization, this method bypasses policy optimization entirely during the discovery phase to avoid unnecessary overhead. Instead, it employs a tree-search strategy inspired by the "Go-With-The-Winner" algorithm, guided by a measure of epistemic uncertainty to systematically uncover new states. This approach not only explores an order of magnitude more efficiently than standard baselines on difficult Atari benchmarks but also generates high-reward trajectories that can be successfully distilled into robust executable policies using supervised backward learning.