mlsanityData to 7 Oct 2026

WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation

arXiv 2201.05955 ↗First posted 16 Jan 2022cs.CLcited by 200

Not on the map. Why?

Abstract (arXiv)

A recurring challenge of crowdsourcing NLP datasets at scale is that human writers often rely on repetitive patterns when crafting examples, leading to a lack of linguistic diversity. We introduce a novel approach for dataset creation based on worker and AI collaboration, which brings together the generative strength of language models and the evaluative strength of humans. Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns. Machine generated examples are then automatically filtered, and finally revised and labeled by human crowdworkers. The resulting dataset, WANLI, consists of 107,885 NLI examples and presents unique empirical strengths over existing NLI datasets. Remarkably, training a model on WANLI improves performance on eight out-of-domain test sets we consider, including by 11% on HANS and 9% on Adversarial NLI, compared to training on the 4x larger MultiNLI. Moreover, it continues to be more effective than MultiNLI augmented with other NLI datasets. Our results demonstrate the promise of leveraging natural language generation techniques and re-imagining the role of humans in the dataset creation process.

Key facts. Cited by 200 papers in the map (data through 7 Oct 2026). Builds on 56 of its 86 references that are in the map. First posted on arXiv 16 Jan 2022.

Builds on

56 of 86 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
38 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  2. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  3. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  10. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  11. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  12. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  13. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  14. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  15. 2018Hypothesis Only Baselines in Natural Language Inferencecited by 291
  16. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  17. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  18. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  19. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  20. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  21. 2021Generating Datasets with Pretrained Language Modelscited by 192
  22. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  23. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  24. 2020Adversarial Filters of Dataset Biasescited by 159
  25. 2018Breaking NLI Systems with Sentences that Require Simple Lexical Inferencescited by 134
  26. 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 133
  27. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  28. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  29. 2021Competency Problems: On Finding and Removing Artifacts in Language Datacited by 91
  30. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  31. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 71
  32. 2021Tailor: Generating and Perturbing Text with Semantic Controlscited by 67
  33. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  34. 2020Generating Counter Narratives against Online Hate Speech: Data and Strategiescited by 58
  35. 2021Neural Data Augmentation via Example Extrapolationcited by 54
  36. 2019Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasetscited by 46
  37. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  38. 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 80; 2024: 49; 2025: 29; 2026: 8.

200 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  4. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  5. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  6. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  7. 2022Self-critiquing models for assisting human evaluatorscited by 372
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  10. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  11. 2022Is GPT-3 a Good Data Annotator?cited by 208
  12. 2022Can Foundation Models Wrangle Your Data?cited by 173
38 more
  1. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  2. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  3. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  4. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  5. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  6. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  7. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  8. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  9. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  10. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  11. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  12. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
  13. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  14. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  15. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  16. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 78
  17. 2023Cognitive Reframing of Negative Thoughts through Human-Language Model Interactioncited by 77
  18. 2023LLMs Accelerate Annotation for Medical Information Extractioncited by 77
  19. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  20. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  21. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 68
  22. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  23. 2022Keep Me Updated! Memory Management in Long-term Conversationscited by 62
  24. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  25. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 61
  26. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Codecited by 61
  27. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  28. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 56
  29. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  30. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  31. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  32. 2023SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tablescited by 51
  33. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 50
  34. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  35. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  36. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  37. 2022Controllable Dialogue Simulation with In-Context Learningcited by 42
  38. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42

50 most cited of 200 All 200 (JSON)