mlsanityData to 7 Oct 2026

PAWS: Paraphrase Adversaries from Word Scrambling

arXiv 1904.01130 ↗First posted 1 Apr 2019cs.CLcited by 395

Not on the map. Why?

Abstract (arXiv)

Existing paraphrase identification datasets lack sentence pairs that have high lexical overlap without being paraphrases. Models trained on such data fail to distinguish pairs like flights from New York to Florida and flights from Florida to New York. This paper introduces PAWS (Paraphrase Adversaries from Word Scrambling), a new dataset with 108,463 well-formed paraphrase and non-paraphrase pairs with high lexical overlap. Challenging pairs are generated by controlled word swapping and back translation, followed by fluency and paraphrase judgments by human raters. State-of-the-art models trained on existing datasets have dismal performance on PAWS (<40% accuracy); however, including PAWS training data for these models improves their accuracy to 85% while maintaining performance on existing tasks. In contrast, models that do not capture non-local contextual information fail even with PAWS training examples. As such, PAWS provides an effective instrument for driving further progress on models that better exploit structure, context, and pairwise comparisons.

Key facts. Cited by 395 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 64; 2022: 88; 2023: 104; 2024: 66; 2025: 39; 2026: 34.

395 papers.

Sort:

By area:

  1. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2023A Comprehensive Overview of Large Language Modelscited by 639
  4. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  5. 2021MetaICL: Learning to Learn In Contextcited by 450
  6. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  8. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  9. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  10. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  11. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  12. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
38 more
  1. 2022Prompting GPT-3 To Be Reliablecited by 258
  2. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  3. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
  4. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  5. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  6. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  7. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  8. 2023Conformal Language Modelingcited by 163
  9. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  10. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 155
  11. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  12. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  13. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  14. 2024Localizing Task Information for Improved Model Merging and Compressioncited by 130
  15. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  16. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  17. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  18. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  19. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  20. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  21. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  22. 2022Language Models are General-Purpose Interfacescited by 96
  23. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  24. 2022ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Promptscited by 85
  25. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  26. 2023Exploring the Benefits of Training Expert Language Models over Instruction Tuningcited by 80
  27. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 74
  28. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  29. 2023Symbol tuning improves in-context learning in language modelscited by 72
  30. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  31. 2022On the Relation between Sensitivity and Accuracy in In-context Learningcited by 71
  32. 2024Parameter Competition Balancing for Model Mergingcited by 71
  33. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  34. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  35. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  36. 2022Investigating Selective Prediction Approaches Across Several Tasks in IID, OOD, and Adversarial Settingscited by 66
  37. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  38. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63

50 most cited of 395 All 395 (JSON)