Surprising properties of dropout in deep networks
Abstract
We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our results
expose surprising differences between the behavior of dropout and more traditional regularizers like
weight decay. For example, on some simple data sets dropout training produces negative weights
even though the output is the sum of the inputs. This provides a counterpoint to the suggestion that
dropout discourages co-adaptation of weights. We also show that the dropout penalty can grow
exponentially in the depth of the network while the weight-decay penalty remains essentially linear,
and that dropout is insensitive to various re-scalings of the input features, outputs, and network
weights. This last insensitivity implies that there are no isolated local minima of the dropout training
criterion. Our work uncovers new properties of dropout,