How Does Noise Help Robustness? Explanation and Exploration under the Neural SDE Framework

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the traditional ResNet structure. However, the resulting ODE system is often unstable---a small input perturbation will be amplified through the ODE system and could eventually blow up. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) which injects random noise by forming a stochastic differential equation. Our framework can model different noise injection regularization techniques in discrete networks, such as dropout and additive/multiplicative noise injection at each block. We provide a theoretical analysis showing the improved robustness of Neural SDE against small input perturbations. Furthermore, we show that the Neural SDE framework can achieve better generalization error than Neural ODE on real datasets and is more stable to small adversarial and non-adversarial input perturbations in practice.

• Machine Intelligence