# PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification

arXiv: https://arxiv.org/abs/1908.11828 · HTML: https://mlsanity.com/paper/1908.11828/

## Key facts

- Cited by 264 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2019.

## Built on by (most cited first, 30 of 264)

- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation](https://mlsanity.com/paper/2107.02137/) (2021)
- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting](https://mlsanity.com/paper/2305.07004/) (2023)
- [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021)
- [Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence](https://mlsanity.com/paper/2404.05892/) (2024)
- [On the Representation Collapse of Sparse Mixture of Experts](https://mlsanity.com/paper/2204.09179/) (2022)
- [UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining](https://mlsanity.com/paper/2304.09151/) (2023)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://mlsanity.com/paper/2310.08754/) (2023)
- [Measure and Improve Robustness in NLP Models: A Survey](https://mlsanity.com/paper/2112.08313/) (2021)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning](https://mlsanity.com/paper/2111.10603/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [Do Multilingual Language Models Think Better in English?](https://mlsanity.com/paper/2308.01223/) (2023)
- [Spanish Pre-trained BERT Model and Evaluation Data](https://mlsanity.com/paper/2308.02976/) (2023)
- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model](https://mlsanity.com/paper/2208.01448/) (2022)
- [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://mlsanity.com/paper/2311.07463/) (2023)

## Abstract (arXiv)

Most existing work on adversarial data generation focuses on English. For example, PAWS (Paraphrase Adversaries from Word Scrambling) consists of challenging English paraphrase identification pairs from Wikipedia and Quora. We remedy this gap with PAWS-X, a new dataset of 23,659 human translated PAWS evaluation pairs in six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. We provide baseline numbers for three models with different capacity to capture non-local context and sentence structure, and using different multilingual training and evaluation regimes. Multilingual BERT fine-tuned on PAWS English plus machine-translated data performs the best, with a range of 83.1-90.8 accuracy across the non-English languages and an average accuracy gain of 23% over the next best model. PAWS-X shows the effectiveness of deep, multilingual pre-training while also leaving considerable headroom as a new challenge to drive multilingual research that better captures structure and contextual information.