mlsanityData to 14 Oct 2026

Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Models

arXiv 2305.12544 ↗First posted 21 May 2023cs.CL cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Recent progress in large language models (LLMs) has enabled the deployment of many generative NLP applications. At the same time, it has also led to a misleading public discourse that ``it's all been solved.'' Not surprisingly, this has, in turn, made many NLP researchers -- especially those at the beginning of their careers -- worry about what NLP research area they should focus on. Has it all been solved, or what remaining questions can we work on regardless of LLMs? To address this question, this paper compiles NLP research directions rich for exploration. We identify fourteen different research areas encompassing 45 research directions that require new research and are not directly solvable by LLMs. While we identify many research areas, many others exist; we do not cover areas currently addressed by LLMs, but where LLMs lag behind in performance or those focused on LLM development. We welcome suggestions for other research directions to include: https://bit.ly/nlp-era-llm

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 265 of its 529 references that are in the map. First posted on arXiv 21 May 2023.

Builds on

265 of 529 references.

References · 265

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  11. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  12. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
38 more
  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  13. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  14. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  15. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  16. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  17. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  18. 2015VQA: Visual Question Answeringcited by 2,506
  19. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  20. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  21. 2020Extracting Training Data from Large Language Modelscited by 2,189
  22. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  23. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  24. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  25. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  26. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  27. 2019Language Models as Knowledge Bases?cited by 1,936
  28. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  29. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  30. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  31. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  32. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  33. 2019Towards VQA Models That Can Readcited by 1,718
  34. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  35. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  36. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  37. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  38. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312

50 most cited of 265 All 265 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2; 2025: 1.

7 papers.

  1. 2024Content Moderation by LLM: From Accuracy to Legitimacycited by 30
  2. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  3. 2024A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciencescited by 5
  4. 2023LMTuner: An user-friendly and highly-integrable Training Framework for fine-tuning Large Language Modelscited by 2
  5. 2025Improving Low-Resource Translation with Dictionary-Guided Fine-Tuning and RL: A Spanish-to-Wayuunaiki Studycited by 1
  6. 2023A Survey of Spanish Clinical Language Modelscited by 0
  7. 2023"A Nova Eletricidade: Aplicacc~oes, Riscos e Tendencias da IA Moderna -- "The New Electricity": Applications, Risks, and Trends in Current AIcited by 0