mlsanityData to 7 Oct 2026

Implicit Representations of Meaning in Neural Language Models

arXiv 2106.00737 ↗First posted 1 Jun 2021cs.CLcited by 179

Not on the map. Why?

Abstract (arXiv)

Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe? In BART and T5 transformer language models, we identify contextual word representations that function as models of entities and situations as they evolve throughout a discourse. These neural representations have functional similarities to linguistic models of dynamic semantics: they support a linear readout of each entity's current properties and relations, and can be manipulated with predictable effects on language generation. Our results indicate that prediction in pretrained neural language models is supported, at least in part, by dynamic representations of meaning and implicit simulation of entity state, and that this behavior can be learned with only text as training data. Code and data are available at https://github.com/belindal/state-probes .

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 17 of its 35 references that are in the map. First posted on arXiv 1 Jun 2021.

Builds on

17 of 35 references.

References · 17

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  3. 2018Deep contextualized word representationscited by 2,285
  4. 2019Language Models as Knowledge Bases?cited by 1,922
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  8. 2019Designing and Interpreting Probes with Control Taskscited by 608
  9. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  10. 2019Revealing the Dark Secrets of BERTcited by 300
  11. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  12. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
5 more
  1. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  2. 2020Pareto Probing: Trading Off Accuracy for Complexitycited by 55
  3. 2016Simpler Context-Dependent Logical Forms via Model Projectionscited by 33
  4. 2020Infusing Finetuning with Semantic Dependenciescited by 18
  5. 2019Improving Semantic Parsing for Task Oriented Dialogcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 24; 2023: 43; 2024: 47; 2025: 35; 2026: 25.

179 papers.

Sort:

By area:

  1. 2021Program Synthesis with Large Language Modelscited by 3,509
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  6. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  7. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  8. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  9. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  10. 2023Language Models Represent Space and Timecited by 306
  11. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  12. 2023Challenges and Applications of Large Language Modelscited by 251
38 more
  1. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  2. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  3. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  4. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  5. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  6. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  7. 2022Talking About Large Language Modelscited by 174
  8. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  9. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  10. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  11. 2022Linearly Mapping from Image to Text Spacecited by 139
  12. 2022Language Models as Agent Modelscited by 131
  13. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  14. 2022Natural Language to Code Translation with Executioncited by 122
  15. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 118
  16. 2024Evaluating the World Model Implicit in a Generative Modelcited by 107
  17. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  18. 2023How do Language Models Bind Entities in Context?cited by 102
  19. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  20. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  21. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  22. 2022Meaning without reference in large language modelscited by 76
  23. 2023Eight Things to Know about Large Language Modelscited by 73
  24. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  25. 2023Learning to Model the World with Languagecited by 66
  26. 2023Evaluating Spatial Understanding of Large Language Modelscited by 65
  27. 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 63
  28. 2022Do Large Language Models know what humans know?cited by 60
  29. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 59
  30. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 49
  31. 2024ICLR: In-Context Learning of Representationscited by 49
  32. 2024LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editingcited by 47
  33. 2023Personas as a Way to Model Truthfulness in Language Modelscited by 46
  34. 2025What Has a Foundation Model Found? Using Inductive Bias to Probe for World Modelscited by 46
  35. 2023Entity Tracking in Language Modelscited by 44
  36. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  37. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  38. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39

50 most cited of 179 All 179 (JSON)