mlsanityData to 14 Oct 2026

Emergent Representations of Program Semantics in Language Models Trained on Programs

arXiv 2305.11169 ↗First posted 18 May 2023cs.LG cs.AI cs.CLcited by 30

On the map: World models → World models for language agents

Abstract (arXiv)

We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code. Our code is available at https://github.com/charlesjin/emergent-semantics.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 25 of its 47 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

25 of 47 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 885
  9. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  10. 2019Designing and Interpreting Probes with Control Taskscited by 640
  11. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 498
  12. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
13 more
  1. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  2. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  3. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  4. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  5. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 119
  6. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 84
  7. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
  8. 2021Language Model Evaluation Beyond Perplexitycited by 78
  9. 2021Chess as a Testbed for Language Model State Trackingcited by 72
  10. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  11. 2021Latent Execution for Neural Program Synthesiscited by 35
  12. 2017Neural Program Meta-Inductioncited by 23
  13. 2022Evaluating Distributional Distortion in Neural Language Modelingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 6; 2026: 11.

30 papers.

Sort:

By area:

  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 246
  2. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 166
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  4. 2023From task structures to world models: What do LLMs know?cited by 46
  5. 2024What Do Language Models Hear? Probing for Auditory Representations in Language Modelscited by 15
  6. 2024A Theory of Response Sampling in LLMs: Part Descriptive and Part Prescriptivecited by 8
  7. 2025When Do Neural Networks Learn World Models?cited by 8
  8. 2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetrycited by 8
  9. 2024Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Netscited by 6
  10. 2025LLMs Lean on Priors, Not Programming Language Semanticscited by 6
  11. 2024Prompting Explicit and Implicit Knowledge for Multi-hop Question Answering Based on Human Reading Processcited by 4
  12. 2025Probing Neural Topology of Large Language Modelscited by 4
18 more
  1. 2025Tracking World States with Language Models: State-Based Evaluation Using Chesscited by 4
  2. 2026Scale Dependent Data Duplicationcited by 4
  3. 2023Towards Understanding What Code Language Models Learnedcited by 3
  4. 2025Knowing the Facts but Choosing the Shortcut: Understanding How Large Language Models Compare Entitiescited by 2
  5. 2024Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Datacited by 1
  6. 2024Unveiling the Decision-Making Process in Reinforcement Learning with Genetic Programmingcited by 1
  7. 2025Can LLMs Learn to Map the World from Local Descriptions?cited by 1
  8. 2026Do multimodal models imagine electric sheep?cited by 1
  9. 2026Latent Programming Horizons in Coding Agentscited by 1
  10. 2026On the Robustness of LLMs' Internal Representation of Code Correctnesscited by 1
  11. 2024Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Samplingcited by 0
  12. 2024Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Predictioncited by 0
  13. 2026Understanding the Emergence of Seemingly Useless Features in Next-Token Predictorscited by 0
  14. 2026LaGO: Latent Action Guidance for Online Reinforcement Learningcited by 0
  15. 2026When Does Reward Teach State? A Hidden-Automaton Instrument and a Group-Language Warning Signalcited by 0
  16. 2026Future Querying: Can LLMs Serve as Implicit Medical World Models?cited by 0
  17. 2026ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Modelscited by 0
  18. 2026LLMs program using Reusable and Composable Abstract Representationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Language Models Struggle to Use Representations Learned In-Context
  2. 2024States Hidden in Hidden States: Implicit Discrete State Representations Emerge in LLMs' Hidden States
  3. 2026What do Language Models Learn and When? The Implicit Curriculum Hypothesis
  4. 2026Are you Synthesizing or Recalling? Evaluating LLMs on Algorithmic Code Retrieval
  5. 2026Towards Understanding What State Space Models Learn About Code
  6. 2025A Code Comprehension Benchmark for Large Language Models for Code
  7. 2026LLMs Explain't: A Post-Mortem on Semantic Interpretability in Transformer Models
  8. 2025Enabling Global, Human-Centered Explanations for LLMs:From Tokens to Interpretable Code and Test Generation