# Does Putting a Linguist in the Loop Improve NLU Data Collection?

arXiv: https://arxiv.org/abs/2104.07179 · HTML: https://mlsanity.com/paper/2104.07179/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 26 of 26)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [Adversarial NLI: A New Benchmark for Natural Language Understanding](https://mlsanity.com/paper/1910.14599/) (2019)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [Revisiting Few-sample BERT Fine-tuning](https://mlsanity.com/paper/2006.05987/) (2020)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://mlsanity.com/paper/2004.11999/) (2020)
- [OCNLI: Original Chinese Natural Language Inference](https://mlsanity.com/paper/2010.05444/) (2020)
- [UnNatural Language Inference](https://mlsanity.com/paper/2101.00010/) (2020)
- [A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk](https://mlsanity.com/paper/1712.05796/) (2017)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [ANLIzing the Adversarial Natural Language Inference Dataset](https://mlsanity.com/paper/2010.12729/) (2020)
- [HELP: A Dataset for Identifying Shortcomings of Neural Models in Monotonicity Reasoning](https://mlsanity.com/paper/1904.12166/) (2019)
- [Controlled Crowdsourcing for High-Quality QA-SRL Annotation](https://mlsanity.com/paper/1911.03243/) (2019)
- [Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options](https://mlsanity.com/paper/2010.06122/) (2020)
- [New Protocols and Negative Results for Textual Entailment Data Collection](https://mlsanity.com/paper/2004.11997/) (2020)

## Built on by (most cited first, 29 of 29)

- [Fact-Checking Complex Claims with Program-Guided Reasoning](https://mlsanity.com/paper/2305.12744/) (2023)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches](https://mlsanity.com/paper/2211.16285/) (2022)
- [FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction](https://mlsanity.com/paper/2403.02270/) (2024)
- [RuCoLA: Russian Corpus of Linguistic Acceptability](https://mlsanity.com/paper/2210.12814/) (2022)
- [Analyzing Dynamic Adversarial Training Data in the Limit](https://mlsanity.com/paper/2110.08514/) (2021)
- [Building Efficient Universal Classifiers with Natural Language Inference](https://mlsanity.com/paper/2312.17543/) (2023)
- [Ev2R: Evaluating Evidence Retrieval in Automated Fact-Checking](https://mlsanity.com/paper/2411.05375/) (2024)
- [Localizing Factual Inconsistencies in Attributable Text Generation](https://mlsanity.com/paper/2410.07473/) (2024)
- [With a Little Push, NLI Models can Robustly and Efficiently Predict Faithfulness](https://mlsanity.com/paper/2305.16819/) (2023)
- [NLI under the Microscope: What Atomic Hypothesis Decomposition Reveals](https://mlsanity.com/paper/2502.08080/) (2025)
- [Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks](https://mlsanity.com/paper/2204.10615/) (2022)
- [Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasis](https://mlsanity.com/paper/2411.19655/) (2024)
- [WeCheck: Strong Factual Consistency Checker via Weakly Supervised Learning](https://mlsanity.com/paper/2212.10057/) (2022)
- [Linguistic Blind Spots of Large Language Models](https://mlsanity.com/paper/2503.19260/) (2025)
- [Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biases](https://mlsanity.com/paper/2305.18917/) (2023)
- [No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference](https://mlsanity.com/paper/2306.09918/) (2023)
- [SpaceNLI: Evaluating the Consistency of Predicting Inferences in Space](https://mlsanity.com/paper/2307.02269/) (2023)
- [Explicating the Implicit: Argument Detection Beyond Sentence Boundaries](https://mlsanity.com/paper/2408.04246/) (2024)
- [A Graph-based Verification Framework for Fact-Checking](https://mlsanity.com/paper/2503.07282/) (2025)
- [Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering](https://mlsanity.com/paper/2511.07659/) (2025)
- [InferES : A Natural Language Inference Corpus for Spanish Featuring Negation-Based Contrastive and Adversarial Examples](https://mlsanity.com/paper/2210.03068/) (2022)
- [What does the Failure to Reason with "Respectively" in Zero/Few-Shot Settings Tell Us about Language Models?](https://mlsanity.com/paper/2305.19597/) (2023)
- [COLE: a Comprehensive Benchmark for French Language Understanding Evaluation](https://mlsanity.com/paper/2510.05046/) (2025)
- [On Degrees of Freedom in Defining and Testing Natural Language Understanding](https://mlsanity.com/paper/2305.15130/) (2023)
- [BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, Rerankers and LLMs](https://mlsanity.com/paper/2603.11991/) (2026)
- [Zero-shot Factual Consistency Evaluation Across Domains](https://mlsanity.com/paper/2408.04114/) (2024)
- [An Approach for Auto Generation of Labeling Functions for Software Engineering Chatbots](https://mlsanity.com/paper/2410.07094/) (2024)
- [ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims](https://mlsanity.com/paper/2603.26449/) (2026)

## Abstract (arXiv)

Many crowdsourced NLP datasets contain systematic gaps and biases that are identified only after data collection is complete. Identifying these issues from early data samples during crowdsourcing should make mitigation more efficient, especially when done iteratively. We take natural language inference as a test case and ask whether it is beneficial to put a linguist `in the loop' during data collection to dynamically identify and address gaps in the data by introducing novel constraints on the task. We directly compare three data collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop intervention with iteratively-updated constraints on the task, and (iii) an extension of linguist-in-the-loop that provides direct interaction between linguists and crowdworkers via a chatroom. The datasets collected with linguist involvement are more reliably challenging than baseline, without loss of quality. But we see no evidence that using this data in training leads to better out-of-domain model performance, and the addition of a chat platform has no measurable effect on the resulting dataset. We suggest integrating expert analysis textitduring data collection so that the expert can dynamically address gaps and biases in the dataset.