mlsanityData to 14 Oct 2026

The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail

arXiv 2110.08300 ↗First posted 15 Oct 2021cs.CL cs.AIcited by 38

Not on the map. Why?

Abstract (arXiv)

Researchers in NLP often frame and discuss research results in ways that serve to deemphasize the field's successes, often in response to the field's widespread hype. Though well-meaning, this has yielded many misleading or false claims about the limits of our best technology. This is a problem, and it may be more serious than it looks: It harms our credibility in ways that can make it harder to mitigate present-day harms, like those involving biased systems for content moderation or resume screening. It also limits our ability to prepare for the potentially enormous impacts of more distant future advances. This paper urges researchers to be careful about these claims and suggests some research directions and communication strategies that will make it easier to avoid or rebut them.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 75 of its 112 references that are in the map. First posted on arXiv 15 Oct 2021.

Builds on

75 of 112 references.

References · 75

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  12. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
38 more
  1. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  3. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  6. 2020Aligning AI With Shared Human Valuescited by 723
  7. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  8. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 571
  9. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  10. 2018Gender Bias in Coreference Resolutioncited by 442
  11. 2018AI safety via debatecited by 400
  12. 2016Cooperative Inverse Reinforcement Learningcited by 300
  13. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  14. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2019Adversarial Attacks and Defenses in Images, Graphs and Text: A Reviewcited by 237
  17. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  18. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
  19. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 169
  20. 2021Alignment of Language Agentscited by 168
  21. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 165
  22. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  23. 2020Adversarial Filters of Dataset Biasescited by 160
  24. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  25. 2020With Little Power Comes Great Responsibilitycited by 142
  26. 2021The Benchmark Lotterycited by 136
  27. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  28. 2018Achieving Human Parity on Automatic Chinese to English News Translationcited by 119
  29. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  30. 2019Semantics-aware BERT for Language Understandingcited by 112
  31. 2020Adversarial Training for Large Neural Language Modelscited by 111
  32. 2020When Do You Need Billions of Words of Pretraining Data?cited by 104
  33. 2018Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluationcited by 88
  34. 2020Reevaluating Adversarial Examples in Natural Languagecited by 85
  35. 2020Mind the Trade-off: Debiasing NLU Models without Degrading the In-distribution Performancecited by 85
  36. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  37. 2021Low-Complexity Probing via Finding Subnetworkscited by 64
  38. 2018Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translationcited by 58

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 14; 2023: 16; 2024: 4; 2025: 1; 2026: 3.

38 papers.

Sort:

By area:

  1. 2022Active Example Selection for In-Context Learningcited by 196
  2. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  3. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  4. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  5. 2024Visibility into AI Agentscited by 79
  6. 2023Eight Things to Know about Large Language Modelscited by 73
  7. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  8. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 51
  9. 2022Can language models handle recursively nested grammatical structures? A case study on comparing models and humanscited by 41
  10. 2023Dissociating language and thought in large language modelscited by 29
  11. 2023Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understandingcited by 20
  12. 2022On the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlationscited by 19
26 more
  1. 2022The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learningcited by 17
  2. 2023SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Modelscited by 15
  3. 2022Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learningcited by 13
  4. 2022Lexical Generalization Improves with Larger Models and Longer Trainingcited by 9
  5. 2022longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attackscited by 8
  6. 2022Cross-lingual Similarity of Multilingual Representations Revisitedcited by 8
  7. 2023Whispers of Doubt Amidst Echoes of Triumph in NLP Robustnesscited by 5
  8. 2022Did AI get more negative recently?cited by 4
  9. 2023Understanding Natural Language Understanding Systems. A Critical Analysiscited by 4
  10. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluationcited by 4
  11. 2024Questionable practices in machine learningcited by 4
  12. 2023NLLG Quarterly arXiv Report 06/23: What are the most influential current AI Papers?cited by 3
  13. 2023Artificial Intelligence and Aesthetic Judgmentcited by 3
  14. 2022Exploring Fake News Detection with Heterogeneous Social Media Context Graphscited by 2
  15. 2023Ranger: A Toolkit for Effect-Size Based Multi-Task Evaluationcited by 2
  16. 2022Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarkscited by 1
  17. 2022A Major Obstacle for NLP Research: Let's Talk about Time Allocation!cited by 1
  18. 2023Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditingcited by 1
  19. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  20. 2024Do Large Language Models Advocate for Inferentialism?cited by 1
  21. 2022Integrating Linguistic Theory and Neural Language Modelscited by 0
  22. 2023Measuring Adversarial Datasetscited by 0
  23. 2025Practising responsibility: Ethics in NLP as a hands-on coursecited by 0
  24. 2026Toward Agentic Governance: What Shapes LLM-Agent Intervention in Public Forums?cited by 0
  25. 2026The Almost Intelligent Revolution: Options for Scaling Up Deliberation and Empowering People with AIcited by 0
  26. 2026Deposon: An Auditable, Conservation-Guaranteed, Game-Theoretically Tested Scattering Layer over LLM Reasoning Pathscited by 0