mlsanityData to 7 Oct 2026

Language Models Represent Space and Time

arXiv 2310.02207 ↗First posted 3 Oct 2023cs.LG cs.AI cs.CLcited by 306

Not on the map. Why?

Cited by 35 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.

Key facts. Cited by 306 papers in the map (data through 7 Oct 2026). Builds on 32 of its 56 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

32 of 56 references.

References · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  9. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  11. 2022Mass-Editing Memory in a Transformercited by 976
  12. 2022Toy Models of Superpositioncited by 928
20 more
  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  5. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  6. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  7. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  8. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  9. 2020Experience Grounds Languagecited by 241
  10. 2023An Overview of Catastrophic AI Riskscited by 214
  11. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  12. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  13. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  14. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  15. 2023The Quantization Model of Neural Scalingcited by 137
  16. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  17. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  18. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  19. 2021Chess as a Testbed for Language Model State Trackingcited by 69
  20. 2021Do Language Models Know the Way to Rome?cited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 78; 2025: 104; 2026: 114.

306 papers.

Sort:

By area:

  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  4. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  5. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  6. 2023Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectivescited by 253
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  8. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  9. 2024Programming Refusal with Conditional Activation Steeringcited by 191
  10. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  11. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithfulcited by 166
  12. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
38 more
  1. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  2. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  3. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  4. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  5. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  6. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 131
  7. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  8. 2024Large Language Models for Time Series: A Surveycited by 113
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  10. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  11. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  12. 2024Universal Neurons in GPT2 Language Modelscited by 100
  13. 2023On Generative Agents in Recommendationcited by 88
  14. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  15. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 83
  16. 2024Robust agents learn causal world modelscited by 81
  17. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  18. 2025The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemscited by 57
  19. 2024The Geometry of Concepts: Sparse Autoencoder Feature Structurecited by 52
  20. 2024ICLR: In-Context Learning of Representationscited by 49
  21. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  22. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  23. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 46
  24. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 44
  25. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
  26. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 39
  27. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  28. 2024LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling lawcited by 32
  29. 2024On the Challenges and Opportunities in Generative AIcited by 31
  30. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 31
  31. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 30
  32. 2024Language Models Represent Beliefs of Self and Otherscited by 29
  33. 2024CityGPT: Empowering Urban Spatial Cognition of Large Language Modelscited by 29
  34. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  35. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  36. 2025Towards Understanding Distilled Reasoning Models: A Representational Approachcited by 26
  37. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  38. 2024AgentMove: A Large Language Model based Agentic Framework for Zero-shot Next Location Predictioncited by 25

50 most cited of 306 All 306 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
  2. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
  3. 2026From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs
  4. 2024Can Large Language Models Create New Knowledge for Spatial Reasoning Tasks?
  5. 2026Do LLMs Build World Models From Text? A Multilingual Diagnostic of Spatial Reasoning
  6. 2025LTD-Bench: Evaluating Large Language Models by Letting Them Draw
  7. 2023Are Large Language Models Temporally Grounded?
  8. 2024Do Language Models Understand Time?