# What Limits Us? Analyzing Self-Reported Limitations in NLP Research

arXiv: https://arxiv.org/abs/2609.15191 · HTML: https://mlsanity.com/paper/2609.15191/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 11 of 11)

- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysis](https://mlsanity.com/paper/2310.15100/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Papers](https://mlsanity.com/paper/2507.02694/) (2025)
- [The ACL OCL Corpus: Advancing Open Science in Computational Linguistics](https://mlsanity.com/paper/2305.14996/) (2023)
- [LimGen: Probing the LLMs for Generating Suggestive Limitations of Research Papers](https://mlsanity.com/paper/2403.15529/) (2024)
- [LimTopic: LLM-based Topic Modeling and Text Summarization for Analyzing Scientific Articles limitations](https://mlsanity.com/paper/2503.10658/) (2025)
- [HICode: Hierarchical Inductive Coding with LLMs](https://mlsanity.com/paper/2509.17946/) (2025)
- [Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigor](https://mlsanity.com/paper/2506.14652/) (2025)

## Abstract (arXiv)

Since late 2022, a Limitations section has become mandatory at many top-tier NLP conferences. The growing number of accepted papers at these venues has resulted in a vast corpus of self-reported limitations that cannot all be manually reviewed, yet remains systematically unanalyzed. Therefore, in this paper, we conduct a large-scale analysis of the Limitations sections from ACL and EMNLP papers published between 2020 and 2025 to understand what researchers disclose about their own work. To do so, we implement a novel human-AI framework for iterative hybrid qualitative coding. This framework enables us to investigate trends in self-reported limitations over time, their correlations with specific paper attributes, and the writing patterns that recur around these disclosures. Our findings offer a critical reflection on the diverse reported challenges as well as the self-reporting practices of researchers in the NLP community.