mlsanityData to 7 Oct 2026

Language Model Behavior: A Comprehensive Survey

arXiv 2303.11504 ↗First posted 20 Mar 2023cs.CLcited by 92

Not on the map. Why?

Abstract (arXiv)

Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about large language model capabilities, thus providing a resource for applied work and for research in adjacent fields that use language models.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 285 of its 393 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

285 of 393 references.

References · 285

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  11. 2020Scaling Laws for Neural Language Modelscited by 6,411
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
38 more
  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  13. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  14. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  15. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  16. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  17. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  18. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  19. 2022Emergent Abilities of Large Language Modelscited by 2,483
  20. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  21. 2019Language Models as Knowledge Bases?cited by 1,922
  22. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  23. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  24. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  25. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  26. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  27. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  28. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  29. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  30. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  31. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  32. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  33. 2018Self-Attention with Relative Position Representationscited by 1,112
  34. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  35. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  36. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  37. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  38. 2022Red Teaming Language Models with Language Modelscited by 983

50 most cited of 285 All 285 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 22; 2024: 24; 2025: 34; 2026: 12.

92 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  2. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  3. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  4. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  5. 2024Do LLMs write like humans? Variation in grammatical and rhetorical stylescited by 62
  6. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  7. 2023Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directionscited by 55
  8. 2024Auxiliary task demands mask the capabilities of smaller language modelscited by 54
  9. 2023Do LLM Agents Exhibit Social Behavior?cited by 53
  10. 2024People cannot distinguish GPT-4 from a human in a Turing testcited by 46
  11. 2024Scalable Influence and Fact Tracing for Large Language Model Pretrainingcited by 40
  12. 2023Does GPT-4 pass the Turing test?cited by 37
38 more
  1. 2024Benchmarking Knowledge Boundary for Large Language Models: A Different Perspective on Model Evaluationcited by 37
  2. 2023HowToCaption: Prompting LLMs to Transform Video Annotations at Scalecited by 34
  3. 2023Distilled GPT for Source Code Summarizationcited by 30
  4. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  5. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  6. 2023Inductive reasoning in humans and large language modelscited by 23
  7. 2024(Ir)rationality and Cognitive Biases in Large Language Modelscited by 23
  8. 2023Position: Key Claims in LLM Research Have a Long Tail of Footnotescited by 20
  9. 2024Causal Estimation of Memorisation Profilescited by 20
  10. 2023Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?cited by 19
  11. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 18
  12. 2023Machine Psychologycited by 16
  13. 2024What Drives Performance in Multilingual Language Models?cited by 14
  14. 2025Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scalecited by 13
  15. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  16. 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
  17. 2024The Factuality of Large Language Models in the Legal Domaincited by 10
  18. 2024Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topicscited by 9
  19. 2024Natural Language Processing RELIES on Linguisticscited by 9
  20. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  21. 2024A systematic investigation of learnability from single child linguistic inputcited by 8
  22. 2023Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blindscited by 7
  23. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  24. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic reviewcited by 7
  25. 2025EasyEdit2: An Easy-to-use Steering Framework for Editing Large Language Modelscited by 7
  26. 2025Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessmentcited by 7
  27. 2025Do language models accommodate their users? A study of linguistic convergencecited by 7
  28. 2025Linguistic Characteristics of AI-Generated Text: A Surveycited by 7
  29. 2023Several categories of Large Language Models (LLMs): A Short Surveycited by 6
  30. 2024Elucidating the design space of language models for image generationcited by 6
  31. 2023Statistical Knowledge Assessment for Large Language Modelscited by 5
  32. 2023Multilingual Nonce Dependency Treebanks: Understanding how Language Models represent and process syntactic structurecited by 5
  33. 2024Fine-Tuning Large Language Models for Scientific Text Classification: A Comparative Studycited by 5
  34. 2025Aligned Probing: Relating Toxic Behavior and Model Internalscited by 5
  35. 2025Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future to eDemorcacycited by 5
  36. 2026Latent Planning Emerges with Scalecited by 5
  37. 2024Achieving Peak Performance for Large Language Models: A Systematic Reviewcited by 4
  38. 2024Is Cognition Consistent with Perception? Assessing and Mitigating Multimodal Knowledge Conflicts in Document Understandingcited by 4

50 most cited of 92 All 92 (JSON)