Multitask Training with Text Data for End-to-End Speech Recognition

We propose a multitask training method for attention-basedend-to-end speech recognition models. We regularize the de-coder in a listen, attend, and spell model by multitask trainingon both audio-text and text-only data. Trained on the 100-hoursubset of LibriSpeech, the proposed method leads to an 11%relative performance improvement over the baseline and is com-parable to language model shallow fusion, without requiring anadditional neural network during decoding. We observe a simi-lar trend on the whole 960-hour LibriSpeech training set. Anal-yses of sample output sentences demonstrate that the proposedmethod can incorporate language level information, suggestingits effectiveness in real-world applications

• Speech Processing