Mean Field Games Flock! The Reinforcement Learning Way

We present a method to enable a large number of agents to learn simultaneously how to flock, which is a natural mass behavior observed in large populations of animals such as birds or fishes. Although this problem has drawn a lot of attention in the literature, it requires many assumptions (e.g. about the environment topology) and can only find tractable solutions in relatively small dimensions. We place ourselves in the framework of Mean Field Games (MFG), where each animal chooses his own acceleration depending on the population behavior. Combining such representation with Deep Reinforcement Learning (DRL) and Normalizing Flows, we obtain a tractable solution requiring very weak assumptions. We show that our algorithm finds a Nash Equilibrium and observe that the agents adapt their own velocities until it matches the crowd's average one. In particular, the induced terminal distribution of agents is stationary in terms of positions and velocities. Our algorithm uses Fictitious Play, and alternates two steps: (1) the computation of an approximate best response with Deep RL, and (2) an estimation of the next distribution of the population using Normalizing Flows. We provide empirical evidences in a multidimensional state space with different (non regular) topologies. Such setting would have been hard to address with standard MFG methods, mostly relying on Partial Differential Equations.

• Machine Intelligence