PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

Most existing work on adversarial data generation focuses only on English. For example, the PAWS (Paraphrase Adversaries from Word Scrambling) dataset consists of English examples for challenging paraphrase identification from Wikipedia and Quora. We remedy this gap with PAWS-X, a new dataset of 23,659 \emph{human} translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacity to capture non-local context and structural word interaction, and using different multilingual training and evaluation regimes. The multilingual BERT model fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an average accuracy gain of 23\% absolute over the best competing model. As such, PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenging benchmark to drive multilingual research that better captures structure and contextual information.

• Natural Language Processing