What is being transferred in transfer learning?

One desired capability for machines is the ability to transfer their understanding of one domain to another domain where data is (usually) scarce. Despite ample adaptation of transfer learning in many deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analysis to address these fundamental questions.
We separate the effect of feature reuse from learning high-level statistics of data and show that some benefit of transfer learning comes from the latter.

• Machine Intelligence