# The Nature of NLP: Analyzing Contributions in NLP Papers

arXiv: https://arxiv.org/abs/2409.19505 · HTML: https://mlsanity.com/paper/2409.19505/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 85 references that are in the map.
- First posted on arXiv 29 Sep 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims](https://mlsanity.com/paper/1906.03538/) (2019)
- [Identification of Tasks, Datasets, Evaluation Metrics, and Numeric Scores for Scientific Leaderboards Construction](https://mlsanity.com/paper/1906.09317/) (2019)
- [Analyzing Dataset Annotation Quality Management in the Wild](https://mlsanity.com/paper/2307.08153/) (2023)
- [TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics](https://mlsanity.com/paper/2101.10273/) (2021)
- [MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting](https://mlsanity.com/paper/2107.00414/) (2021)
- [SciDTB: Discourse Dependency TreeBank for Scientific Abstracts](https://mlsanity.com/paper/1806.03653/) (2018)
- [Overview of the 2021 Key Point Analysis Shared Task](https://mlsanity.com/paper/2110.10577/) (2021)
- [Model Criticism for Long-Form Text Generation](https://mlsanity.com/paper/2210.08444/) (2022)
- [Examining Citations of Natural Language Processing Literature](https://mlsanity.com/paper/2005.00912/) (2020)
- [Every Bite Is an Experience: Key Point Analysis of Business Reviews](https://mlsanity.com/paper/2106.06758/) (2021)
- [A Holistic Framework for Analyzing the COVID-19 Vaccine Debate](https://mlsanity.com/paper/2205.01817/) (2022)
- [SemEval-2021 Task 11: NLPContributionGraph -- Structuring Scholarly NLP Contributions for a Research Knowledge Graph](https://mlsanity.com/paper/2106.07385/) (2021)
- [Evaluating Emotion Arcs Across Languages: Bridging the Global Divide in Sentiment Analysis](https://mlsanity.com/paper/2306.02213/) (2023)
- [A Diachronic Analysis of Paradigm Shifts in NLP Research: When, How, and Why?](https://mlsanity.com/paper/2305.12920/) (2023)
- [Claim Extraction in Biomedical Publications using Deep Discourse Model and Transfer Learning](https://mlsanity.com/paper/1907.00962/) (2019)
- [D3: A Massive Dataset of Scholarly Metadata for Analyzing the State of Computer Science Research](https://mlsanity.com/paper/2204.13384/) (2022)
- [Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards](https://mlsanity.com/paper/2409.12656/) (2024)
- [Read Top News First: A Document Reordering Approach for Multi-Document News Summarization](https://mlsanity.com/paper/2203.10254/) (2022)
- [NLPContributions: An Annotation Scheme for Machine Reading of Scholarly Contributions in Natural Language Processing Literature](https://mlsanity.com/paper/2006.12870/) (2020)
- [A Review of Theory and Practice in Scientometrics](https://mlsanity.com/paper/1501.05462/) (2015)
- [Visualizing a Field of Research: A Methodology of Systematic Scientometric Reviews](https://mlsanity.com/paper/1906.04800/) (2019)

## Built on by (most cited first, 12 of 12)

- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [CHIMERA: A Knowledge Base of Scientific Idea Recombinations for Research Analysis and Ideation](https://mlsanity.com/paper/2505.20779/) (2025)
- [Sci-Reasoning: A Dataset Decoding AI Innovation Patterns](https://mlsanity.com/paper/2601.04577/) (2026)
- [A Position Paper on the Automatic Generation of Machine Learning Leaderboards](https://mlsanity.com/paper/2505.17465/) (2025)
- [ClaimFlow: Tracing the Evolution of Scientific Claims in NLP](https://mlsanity.com/paper/2603.16073/) (2026)
- [Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025](https://mlsanity.com/paper/2606.02255/) (2026)
- [The Rise of AfricaNLP: A Survey of Contributions, Contributors, Community Impact, and Bibliometric Analysis](https://mlsanity.com/paper/2509.25477/) (2025)
- [SciZoom: A Large-scale Benchmark for Hierarchical Scientific Summarization across the LLM Era](https://mlsanity.com/paper/2603.16131/) (2026)
- [Exploring Novelty Differences between Industry and Academia: A Knowledge Entity-centric Perspective](https://mlsanity.com/paper/2603.19319/) (2026)
- [Are Non-English Papers Reviewed Fairly? Language-of-Study Bias in NLP Peer Reviews](https://mlsanity.com/paper/2604.07119/) (2026)
- [The Scientific Contribution Graph: Automated Literature-based Technological Roadmapping at Scale](https://mlsanity.com/paper/2605.15011/) (2026)
- [Eleven Years of BRACIS: A Meta-Scientific Study of the Brazilian Conference on Intelligent Systems](https://mlsanity.com/paper/2608.09964/) (2026)

## Abstract (arXiv)

Natural Language Processing (NLP) is an established and dynamic field. Despite this, what constitutes NLP research remains debated. In this work, we address the question by quantitatively examining NLP research papers. We propose a taxonomy of research contributions and introduce NLPContributions, a dataset of nearly 2k NLP research paper abstracts, carefully annotated to identify scientific contributions and classify their types according to this taxonomy. We also introduce a novel task of automatically identifying contribution statements and classifying their types from research papers. We present experimental results for this task and apply our model to ~29k NLP research papers to analyze their contributions, aiding in the understanding of the nature of NLP research. We show that NLP research has taken a winding path -- with the focus on language and human-centric studies being prominent in the 1970s and 80s, tapering off in the 1990s and 2000s, and starting to rise again since the late 2010s. Alongside this revival, we observe a steady rise in dataset and methodological contributions since the 1990s, such that today, on average, individual NLP papers contribute in more ways than ever before. Our dataset and analyses offer a powerful lens for tracing research trends and offer potential for generating informed, data-driven literature surveys.