Modulating early visual processing by language
Abstract
It is commonly assumed that language refers to high-level visual concepts while
leaving low-level visual processing unaffected. This view dominates the current
literature in computational models for language-vision tasks, where visual and
linguistic input are mostly processed independently before being fused into a single
representation. In this paper, we deviate from this classic pipeline and propose to
modulate the entire visual processing by linguistic input. Specifically, we condition
the batch normalization parameters of a pretrained residual network (ResNet) on a
language embedding. This approach, which we call MOdulated RESnet (MORES),
significantly improves strong baselines on two visual question answering tasks. Our
ablation study shows that modulating from the early stages of the visual processing
is beneficial. We finally show that ResNet image features are effectively grounded.
leaving low-level visual processing unaffected. This view dominates the current
literature in computational models for language-vision tasks, where visual and
linguistic input are mostly processed independently before being fused into a single
representation. In this paper, we deviate from this classic pipeline and propose to
modulate the entire visual processing by linguistic input. Specifically, we condition
the batch normalization parameters of a pretrained residual network (ResNet) on a
language embedding. This approach, which we call MOdulated RESnet (MORES),
significantly improves strong baselines on two visual question answering tasks. Our
ablation study shows that modulating from the early stages of the visual processing
is beneficial. We finally show that ResNet image features are effectively grounded.
Research Areas
