Metrics and continuity in reinforcement learning

Reinforcement learning techniques are being applied to increasingly larger systems where it becomes untenable to maintain direct estimates for
individual states, in particular for continuous-state
systems. Instead, researchers often leverage state
similarity (whether implicitly or explicitly) to
build models that can generalize well from a limited set of samples. The notion of state similarity
used is thus of crucial importance, as it will directly affect the quality of the approximations
and performance of the algorithms. Indeed, there
have been a number of works that investigate –
both on a theoretical and an empirical basis – how
best to construct these neighborhoods and topologies. However, the choice of metric is not always
clear and is often not fully specified when new
algorithms are introduced. In this paper we aim
to clarify the landscape of existing metrics and
provide guidelines for the choice of metric when
designing or implementing algorithms. We do this
by first introducing a unified formalism for specifying these topologies, through the lens of metrics
or distance measures, and clarify the relationship
between them. We establish a hierarchy amongst
the different metrics and their theoretical implications on the Markov Decision Process (MDP)
specifying the reinforcement learning problem.
We complement our theoretical results with empirical evaluations showcasing the differences between the metrics considered.

• Machine Intelligence