mlsanityData to 7 Oct 2026

Eight Things to Know about Large Language Models

arXiv 2304.00612 ↗First posted 2 Apr 2023cs.CL cs.AIcited by 73

Not on the map. Why?

Abstract (arXiv)

The widespread public deployment of large language models (LLMs) in recent months has prompted a wave of new attention and engagement from advocates, policymakers, and scholars from many fields. This attention is a timely response to the many urgent questions that this technology raises, but it can sometimes miss important considerations. This paper surveys the evidence for eight potentially surprising such points: 1. LLMs predictably get more capable with increasing investment, even without targeted innovation. 2. Many important LLM behaviors emerge unpredictably as a byproduct of increasing investment. 3. LLMs often appear to learn and use representations of the outside world. 4. There are no reliable techniques for steering the behavior of LLMs. 5. Experts are not yet able to interpret the inner workings of LLMs. 6. Human performance on a task isn't an upper bound on LLM performance. 7. LLMs need not express the values of their creators nor the values encoded in web text. 8. Brief interactions with LLMs are often misleading.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 78 of its 128 references that are in the map. First posted on arXiv 2 Apr 2023.

Builds on

78 of 128 references.

References · 78

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  13. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  14. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  15. 2016Concrete Problems in AI Safetycited by 1,582
  16. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  17. 2019Attention is not Explanationcited by 916
  18. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  19. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  20. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  21. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  22. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  23. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  24. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  25. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  26. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  27. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  28. 2022Self-critiquing models for assisting human evaluatorscited by 372
  29. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  30. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  31. 2022Teaching language models to support answers with verified quotescited by 290
  32. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 288
  33. 2022Prompting GPT-3 To Be Reliablecited by 258
  34. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 250
  35. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  36. 2023Pretraining Language Models with Human Preferencescited by 245
  37. 2022Data Contamination: From Memorization to Exploitationcited by 234
  38. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 221

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 23; 2025: 9; 2026: 9.

73 papers.

Sort:

By area:

  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  4. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  5. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  6. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  7. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  8. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  9. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  10. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  11. 2024Emergent World Models and Latent Variable Estimation in Chess-Playing Language Modelscited by 60
  12. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
38 more
  1. 2023From task structures to world models: What do LLMs know?cited by 46
  2. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13Bcited by 35
  3. 2023Open-Sourcing Highly Capable Foundation Models: An evaluation of risks, benefits, and alternative methods for pursuing open-source objectivescited by 31
  4. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  5. 2024XAI meets LLMs: A Survey of the Relation between Explainable AI and Large Language Modelscited by 23
  6. 2024Aviary: training language agents on challenging scientific taskscited by 23
  7. 2024The Shutdown Problem: An AI Engineering Puzzle for Decision Theoristscited by 22
  8. 2023What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theorycited by 19
  9. 2023In-Context Learning User Simulators for Task-Oriented Dialog Systemscited by 19
  10. 2023Transformers in Healthcare: A Surveycited by 17
  11. 2023Game of Tones: Faculty detection of GPT-4 generated content in university assessmentscited by 15
  12. 2023Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI)cited by 14
  13. 2023Explainability in AI Policies: A Critical Review of Communications, Reports, Regulations, and Standards in the EU, US, and UKcited by 13
  14. 2023Gene Set Summarization using Large Language Modelscited by 12
  15. 2024ElecBench: a Power Dispatch Evaluation Benchmark for Large Language Modelscited by 12
  16. 2023Towards Publicly Accountable Frontier LLMs: Building an External Scrutiny Ecosystem under the ASPIRE Frameworkcited by 11
  17. 2024Evidence of a log scaling law for political persuasion with large language modelscited by 11
  18. 2024Delving into the Reversal Curse: How Far Can Large Language Models Generalize?cited by 11
  19. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Modelscited by 11
  20. 2025Epistemological Fault Lines Between Human and Artificial Intelligencecited by 11
  21. 2023Tell, don't show: Declarative facts influence how LLMs generalizecited by 8
  22. 2024FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formatscited by 8
  23. 2023WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Modelcited by 6
  24. 2023On the Unexpected Abilities of Large Language Modelscited by 5
  25. 2024From Neurons to Neutrons: A Case Study in Interpretabilitycited by 5
  26. 2024Mechanistic interpretability of large language models with applications to the financial services industrycited by 5
  27. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlookcited by 5
  28. 2023Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Educationcited by 4
  29. 2025This Is Your Doge, If It Please You: Exploring Deception and Robustness in Mixture of LLMscited by 4
  30. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judgecited by 4
  31. 2023Mini-Giants: "Small" Language Models and Open Source Win-Wincited by 3
  32. 2024LLM-based Extraction of Contradictions from Patentscited by 3
  33. 2024The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challengescited by 3
  34. 2025Modeling Motivated Reasoning in Law: Evaluating Strategic Role Conditioning in LLM Summarizationcited by 3
  35. 2023Enhancing Pipeline-Based Conversational Agents with Large Language Modelscited by 2
  36. 2023Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applicationscited by 2
  37. 2024Optimizing Psychological Counseling with Instruction-Tuned Large Language Modelscited by 2
  38. 2024Better Language Models Exhibit Higher Visual Alignmentcited by 2

50 most cited of 73 All 73 (JSON)