mlsanityData to 7 Oct 2026

First Tragedy, then Parse: History Repeats Itself in the New Era of Large Language Models

arXiv 2311.05020 ↗First posted 8 Nov 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Many NLP researchers are experiencing an existential crisis triggered by the astonishing success of ChatGPT and other systems based on large language models (LLMs). After such a disruptive change to our understanding of the field, what is left to do? Taking a historical lens, we look for guidance from the first era of LLMs, which began in 2005 with large n-gram models for machine translation (MT). We identify durable lessons from the first era, and more importantly, we identify evergreen problems where NLP researchers can continue to make meaningful contributions in areas where LLMs are ascendant. We argue that disparities in scale are transient and researchers can work to reduce them; that data, rather than hardware, is still a bottleneck for many applications; that meaningful realistic evaluation is still an open problem; and that there is still room for speculative approaches.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 45 of its 139 references that are in the map. First posted on arXiv 8 Nov 2023.

Builds on

45 of 139 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  12. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
33 more
  1. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  2. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  3. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  4. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  5. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  6. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  7. 2017Six Challenges for Neural Machine Translationcited by 334
  8. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  9. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  10. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  11. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  12. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  13. 2022Compute Trends Across Three Eras of Machine Learningcited by 161
  14. 2020The Hardware Lotterycited by 160
  15. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  16. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  17. 2023Training Language Models with Language Feedback at Scalecited by 125
  18. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  19. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  20. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  21. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  22. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  23. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  24. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73
  25. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
  26. 2023What Comes Next? Evaluating Uncertainty in Neural Text Generators Against Human Production Variabilitycited by 51
  27. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  28. 2018Fluency Over Adequacy: A Pilot Study in Measuring User Trust in Imperfect MTcited by 21
  29. 2022MMTAfrica: Multilingual Machine Translation for African Languagescited by 11
  30. 2020Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignmentcited by 10
  31. 2023Choose Your Weapon: Survival Strategies for Depressed AI Academicscited by 8
  32. 2023"Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision modelscited by 5
  33. 2023Surveying (Dis)Parities and Concerns of Compute Hungry NLP Researchcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2; 2025: 1.

4 papers.

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  2. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  3. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 33
  4. 2025MEQA: A Meta-Evaluation Framework for Question & Answer LLM Benchmarkscited by 2