mlsanityData to 14 Oct 2026

A Philosophical Introduction to Language Models - Part II: The Way Forward

arXiv 2405.03207 ↗First posted 6 May 2024cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

In this paper, the second of two companion pieces, we explore novel philosophical questions raised by recent progress in large language models (LLMs) that go beyond the classical debates covered in the first part. We focus particularly on issues related to interpretability, examining evidence from causal intervention methods about the nature of LLMs' internal representations and computations. We also discuss the implications of multimodal and modular extensions of LLMs, recent debates about whether such systems may meet minimal criteria for consciousness, and concerns about secrecy and reproducibility in LLM research. Finally, we discuss whether LLM-like systems may be relevant to modeling aspects of human cognition, if their architectural characteristics and learning scenario are adequately constrained.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 90 of its 160 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

90 of 160 references.

References · 90

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
38 more
  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  4. 2022Emergent Abilities of Large Language Modelscited by 2,512
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2022Toy Models of Superpositioncited by 971
  11. 2022In-context Learning and Induction Headscited by 964
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  13. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  14. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  15. 2022A Generalist Agentcited by 808
  16. 2017Deep Reinforcement Learning that Matterscited by 754
  17. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  18. 2016The Mythos of Model Interpretabilitycited by 721
  19. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  20. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  21. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  22. 2019Designing and Interpreting Probes with Control Taskscited by 640
  23. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  24. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  25. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  26. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  27. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  28. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  29. 2021Causal Abstractions of Neural Networkscited by 422
  30. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  31. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  32. 2023Large Language Models Fail on Trivial Alterations to Theory-of-Mind Taskscited by 292
  33. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  34. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  35. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  36. 2022Emergent Analogical Reasoning in Large Language Modelscited by 240
  37. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  38. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  2. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objectscited by 13
  3. 2024Transformers Use Causal World Models in Maze-Solving Taskscited by 13
  4. 2025The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Modelscited by 13
  5. 2025From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychologycited by 9
  6. 2024From Frege to chatGPT: Compositionality in language, cognition, and deep neural networkscited by 7
  7. 2025Mechanistic Interpretability Needs Philosophycited by 6
  8. 2026Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networkscited by 6
  9. 2024Philosophy of Cognitive Science in the Age of Deep Learningcited by 5
  10. 2025Are Sparse Autoencoders Useful for Java Function Bug Detection?cited by 2
  11. 2026A Model of Understanding in Deep Learning Systemscited by 2
  12. 2025Self-reflecting Large Language Models: A Hegelian Dialectical Approachcited by 1
6 more
  1. 2025The Outputs of Large Language Models are Meaninglesscited by 1
  2. 2025Representation in large language modelscited by 0
  3. 2025Explaining Neural Networks with Reasonscited by 0
  4. 2025Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligencecited by 0
  5. 2025What do language models model? Transformers, automata, and the format of thoughtcited by 0
  6. 2026Vision-Language Asymmetry in Bistable Image Captioningcited by 0