Training Neural Networks to Produce Compatible Features

This paper makes a first step towards compatible network components. We propose three ways which modify training to make components compatible: (i) We add a shared supervised auxiliary task which discriminates between the common classes. (ii) We add a shared self-supervised auxiliary task: rotation prediction. (iii) We initialize the networks using the same random weights. On CIFAR-10 we show: (i) we can train networks to produce compatible features, without degrading task accuracy compared to training the networks independently. (ii) random initialization has a large effect on compatibility; (ii) we can train incrementally: given previously trained components, we can train new ones which are also compatible with them