mlsanityData to 14 Oct 2026

What Limits Us? Analyzing Self-Reported Limitations in NLP Research

arXiv 2609.15191 ↗First posted 14 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Since late 2022, a Limitations section has become mandatory at many top-tier NLP conferences. The growing number of accepted papers at these venues has resulted in a vast corpus of self-reported limitations that cannot all be manually reviewed, yet remains systematically unanalyzed. Therefore, in this paper, we conduct a large-scale analysis of the Limitations sections from ACL and EMNLP papers published between 2020 and 2025 to understand what researchers disclose about their own work. To do so, we implement a novel human-AI framework for iterative hybrid qualitative coding. This framework enables us to investigate trends in self-reported limitations over time, their correlations with specific paper attributes, and the writing patterns that recur around these disclosures. Our findings offer a critical reflection on the diverse reported challenges as well as the self-reporting practices of researchers in the NLP community.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 30 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

11 of 30 references.

References · 11

  1. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 801
  2. 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
  3. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  4. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  5. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  6. 2025Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Paperscited by 20
  7. 2023The ACL OCL Corpus: Advancing Open Science in Computational Linguisticscited by 17
  8. 2024LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Paperscited by 16
  9. 2025LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitationscited by 12
  10. 2025HICode: Hierarchical Inductive Coding with LLMscited by 12
  11. 2025Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigorcited by 7

Built on by

Not cited yet.