# Emergent Representations of Program Semantics in Language Models Trained on Programs

arXiv: https://arxiv.org/abs/2305.11169 · HTML: https://mlsanity.com/paper/2305.11169/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in World models for language agents, one of 11 directions in World models.
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 18 May 2023.

## Placed in

- World models / [World models for language agents](https://mlsanity.com/wm/world-models-for-language-agents/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [The Debate Over Understanding in AI's Large Language Models](https://mlsanity.com/paper/2210.13966/) (2022)
- [Large Language Models Meet NL2Code: A Survey](https://mlsanity.com/paper/2212.09420/) (2022)
- [Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Color](https://mlsanity.com/paper/2109.06129/) (2021)
- [Leveraging Grammar and Reinforcement Learning for Neural Program Synthesis](https://mlsanity.com/paper/1805.04276/) (2018)
- [Mind's Eye: Grounded Language Model Reasoning through Simulation](https://mlsanity.com/paper/2210.05359/) (2022)
- [Language Model Evaluation Beyond Perplexity](https://mlsanity.com/paper/2106.00085/) (2021)
- [Chess as a Testbed for Language Model State Tracking](https://mlsanity.com/paper/2102.13249/) (2021)
- [Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?](https://mlsanity.com/paper/2104.10809/) (2021)
- [Latent Execution for Neural Program Synthesis](https://mlsanity.com/paper/2107.00101/) (2021)
- [Neural Program Meta-Induction](https://mlsanity.com/paper/1710.04157/) (2017)
- [Evaluating Distributional Distortion in Neural Language Modeling](https://mlsanity.com/paper/2203.12788/) (2022)

## Built on by (most cited first, 30 of 30)

- [Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks](https://mlsanity.com/paper/2307.02477/) (2023)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [From task structures to world models: What do LLMs know?](https://mlsanity.com/paper/2310.04276/) (2023)
- [What Do Language Models Hear? Probing for Auditory Representations in Language Models](https://mlsanity.com/paper/2402.16998/) (2024)
- [A Theory of Response Sampling in LLMs: Part Descriptive and Part Prescriptive](https://mlsanity.com/paper/2402.11005/) (2024)
- [When Do Neural Networks Learn World Models?](https://mlsanity.com/paper/2502.09297/) (2025)
- [Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry](https://mlsanity.com/paper/2601.22588/) (2026)
- [Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets](https://mlsanity.com/paper/2410.01779/) (2024)
- [LLMs Lean on Priors, Not Programming Language Semantics](https://mlsanity.com/paper/2510.03415/) (2025)
- [Prompting Explicit and Implicit Knowledge for Multi-hop Question Answering Based on Human Reading Process](https://mlsanity.com/paper/2402.19350/) (2024)
- [Probing Neural Topology of Large Language Models](https://mlsanity.com/paper/2506.01042/) (2025)
- [Tracking World States with Language Models: State-Based Evaluation Using Chess](https://mlsanity.com/paper/2508.19851/) (2025)
- [Scale Dependent Data Duplication](https://mlsanity.com/paper/2603.06603/) (2026)
- [Towards Understanding What Code Language Models Learned](https://mlsanity.com/paper/2306.11943/) (2023)
- [Knowing the Facts but Choosing the Shortcut: Understanding How Large Language Models Compare Entities](https://mlsanity.com/paper/2510.16815/) (2025)
- [Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data](https://mlsanity.com/paper/2407.13765/) (2024)
- [Unveiling the Decision-Making Process in Reinforcement Learning with Genetic Programming](https://mlsanity.com/paper/2407.14714/) (2024)
- [Can LLMs Learn to Map the World from Local Descriptions?](https://mlsanity.com/paper/2505.20874/) (2025)
- [Do multimodal models imagine electric sheep?](https://mlsanity.com/paper/2605.09693/) (2026)
- [Latent Programming Horizons in Coding Agents](https://mlsanity.com/paper/2607.05188/) (2026)
- [On the Robustness of LLMs' Internal Representation of Code Correctness](https://mlsanity.com/paper/2608.08266/) (2026)
- [Improving Embedding Accuracy for Document Retrieval Using Entity Relationship Maps and Model-Aware Contrastive Sampling](https://mlsanity.com/paper/2410.18105/) (2024)
- [Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction](https://mlsanity.com/paper/2410.18160/) (2024)
- [Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors](https://mlsanity.com/paper/2603.14087/) (2026)
- [LaGO: Latent Action Guidance for Online Reinforcement Learning](https://mlsanity.com/paper/2606.24669/) (2026)
- [When Does Reward Teach State? A Hidden-Automaton Instrument and a Group-Language Warning Signal](https://mlsanity.com/paper/2607.11953/) (2026)
- [Future Querying: Can LLMs Serve as Implicit Medical World Models?](https://mlsanity.com/paper/2608.23248/) (2026)
- [ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models](https://mlsanity.com/paper/2609.36952/) (2026)
- [LLMs program using Reusable and Composable Abstract Representations](https://mlsanity.com/paper/2610.19238/) (2026)

## Abstract (arXiv)

We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code. Our code is available at https://github.com/charlesjin/emergent-semantics.