mlsanityData to 7 Oct 2026

Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?

arXiv 2308.00189 ↗First posted 31 Jul 2023cs.LG cs.AI cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Coaxing out desired behavior from pretrained models, while avoiding undesirable ones, has redefined NLP and is reshaping how we interact with computers. What was once a scientific engineering discipline-in which building blocks are stacked one on top of the other-is arguably already a complex systems science, in which emergent behaviors are sought out to support previously unimagined use cases. Despite the ever increasing number of benchmarks that measure task performance, we lack explanations of what behaviors language models exhibit that allow them to complete these tasks in the first place. We argue for a systematic effort to decompose language model behavior into categories that explain cross-task performance, to guide mechanistic explanations and help future-proof analytic research.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 92 of its 153 references that are in the map. First posted on arXiv 31 Jul 2023.

Builds on

92 of 153 references.

References · 92

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  11. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2020Longformer: The Long-Document Transformercited by 3,362
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  12. 2019Language Models as Knowledge Bases?cited by 1,922
  13. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  14. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  15. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  16. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  17. 2023LIMA: Less Is More for Alignmentcited by 1,080
  18. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  19. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  20. 2022In-context Learning and Induction Headscited by 930
  21. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  22. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  23. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  24. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  25. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  26. 2016The Mythos of Model Interpretabilitycited by 716
  27. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  28. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  29. 2018Object Hallucination in Image Captioningcited by 661
  30. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  31. 2017Interpretable Explanations of Black Boxes by Meaningful Perturbationcited by 611
  32. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  33. 2023Textbooks Are All You Needcited by 561
  34. 2019Defending Against Neural Fake Newscited by 553
  35. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 409
  36. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 373
  37. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  38. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 6; 2025: 6; 2026: 3.

19 papers.

Sort:

By area:

  1. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  2. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  3. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  4. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  5. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  6. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  7. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  8. 2024Evaluating Language Model Agency through Negotiationscited by 43
  9. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 28
  10. 2025EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Treescited by 26
  11. 2024Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Reviewcited by 17
  12. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16
7 more
  1. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
  2. 2025Assessing High-Risk AI Systems under the EU AI Act: From Legal Requirements to Technical Verificationcited by 4
  3. 2025Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learningcited by 2
  4. 2025Prompting as Scientific Inquirycited by 2
  5. 2026Validating Causal Abstraction Metrics on Simulated Complex Systemscited by 0
  6. 2026Harnessing Abundance: A Generativity Perspective on Human-GenAI Collaborationcited by 0
  7. 2026Assessing mentalization in humans and large language modelscited by 0