# To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processing

arXiv: https://arxiv.org/abs/2310.07715 · HTML: https://mlsanity.com/paper/2310.07715/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [The Hardware Lottery](https://mlsanity.com/paper/2009.06489/) (2020)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research](https://mlsanity.com/paper/2010.15581/) (2020)
- [The Gradient of Generative AI Release: Methods and Considerations](https://mlsanity.com/paper/2302.04844/) (2023)
- [Changing the World by Changing the Data](https://mlsanity.com/paper/2105.13947/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [What Do NLP Researchers Believe? Results of the NLP Community Metasurvey](https://mlsanity.com/paper/2208.12852/) (2022)
- [The Elephant in the Room: Analyzing the Presence of Big Tech in Natural Language Processing Research](https://mlsanity.com/paper/2305.02797/) (2023)
- [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021)
- [DyNet: The Dynamic Neural Network Toolkit](https://mlsanity.com/paper/1701.03980/) (2017)
- [Examining Citations of Natural Language Processing Literature](https://mlsanity.com/paper/2005.00912/) (2020)
- [Geographic Citation Gaps in NLP Research](https://mlsanity.com/paper/2210.14424/) (2022)
- [Forgotten Knowledge: Examining the Citational Amnesia in NLP](https://mlsanity.com/paper/2305.18554/) (2023)
- [Should attention be all we need? The epistemic and ethical implications of unification in machine learning](https://mlsanity.com/paper/2205.08377/) (2022)
- [Gender, Productivity, and Prestige in Computer Science Faculty Hiring Networks](https://mlsanity.com/paper/1602.00795/) (2016)

## Built on by (most cited first, 4 of 4)

- [The AI Community Building the Future? A Quantitative Analysis of Development Activity on Hugging Face Hub](https://mlsanity.com/paper/2405.13058/) (2024)
- [Basic Research, Lethal Effects: Military AI Research Funding as Enlistment](https://mlsanity.com/paper/2411.17840/) (2024)
- [Risks of AI-driven product development and strategies for their mitigation](https://mlsanity.com/paper/2506.00047/) (2025)
- [The Algorithmic Gaze of Image Quality Assessment: An Audit and Trace Ethnography of the LAION-Aesthetics Predictor](https://mlsanity.com/paper/2601.09896/) (2026)

## Abstract (arXiv)

NLP is in a period of disruptive change that is impacting our methodologies, funding sources, and public perception. In this work, we seek to understand how to shape our future by better understanding our past. We study factors that shape NLP as a field, including culture, incentives, and infrastructure by conducting long-form interviews with 26 NLP researchers of varying seniority, research area, institution, and social identity. Our interviewees identify cyclical patterns in the field, as well as new shifts without historical parallel, including changes in benchmark culture and software infrastructure. We complement this discussion with quantitative analysis of citation, authorship, and language use in the ACL Anthology over time. We conclude by discussing shared visions, concerns, and hopes for the future of NLP. We hope that this study of our field's past and present can prompt informed discussion of our community's implicit norms and more deliberate action to consciously shape the future.