# A Philosophical Introduction to Language Models - Part II: The Way Forward

arXiv: https://arxiv.org/abs/2405.03207 · HTML: https://mlsanity.com/paper/2405.03207/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 90 of its 160 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 30 of 90)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)

## Built on by (most cited first, 18 of 18)

- [Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space](https://mlsanity.com/paper/2406.19370/) (2024)
- [Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects](https://mlsanity.com/paper/2406.15955/) (2024)
- [Transformers Use Causal World Models in Maze-Solving Tasks](https://mlsanity.com/paper/2412.11867/) (2024)
- [The Benchmarking Epistemology: Validity Theory for Evaluating Machine Learning Models](https://mlsanity.com/paper/2510.23191/) (2025)
- [From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology](https://mlsanity.com/paper/2506.16697/) (2025)
- [From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks](https://mlsanity.com/paper/2405.15164/) (2024)
- [Mechanistic Interpretability Needs Philosophy](https://mlsanity.com/paper/2506.18852/) (2025)
- [Establishing Construct Validity in LLM Capability Benchmarks Requires Nomological Networks](https://mlsanity.com/paper/2603.15121/) (2026)
- [Philosophy of Cognitive Science in the Age of Deep Learning](https://mlsanity.com/paper/2405.04048/) (2024)
- [Are Sparse Autoencoders Useful for Java Function Bug Detection?](https://mlsanity.com/paper/2505.10375/) (2025)
- [A Model of Understanding in Deep Learning Systems](https://mlsanity.com/paper/2604.04171/) (2026)
- [Self-reflecting Large Language Models: A Hegelian Dialectical Approach](https://mlsanity.com/paper/2501.14917/) (2025)
- [The Outputs of Large Language Models are Meaningless](https://mlsanity.com/paper/2509.22206/) (2025)
- [Representation in large language models](https://mlsanity.com/paper/2501.00885/) (2025)
- [Explaining Neural Networks with Reasons](https://mlsanity.com/paper/2505.14424/) (2025)
- [Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence](https://mlsanity.com/paper/2507.22197/) (2025)
- [What do language models model? Transformers, automata, and the format of thought](https://mlsanity.com/paper/2508.18598/) (2025)
- [Vision-Language Asymmetry in Bistable Image Captioning](https://mlsanity.com/paper/2606.08031/) (2026)

## Abstract (arXiv)

In this paper, the second of two companion pieces, we explore novel philosophical questions raised by recent progress in large language models (LLMs) that go beyond the classical debates covered in the first part. We focus particularly on issues related to interpretability, examining evidence from causal intervention methods about the nature of LLMs' internal representations and computations. We also discuss the implications of multimodal and modular extensions of LLMs, recent debates about whether such systems may meet minimal criteria for consciousness, and concerns about secrecy and reproducibility in LLM research. Finally, we discuss whether LLM-like systems may be relevant to modeling aspects of human cognition, if their architectural characteristics and learning scenario are adequately constrained.