mlsanityData to 7 Oct 2026

PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

arXiv 2106.00188 ↗First posted 1 Jun 2021cs.CL cs.AIcited by 47

Not on the map. Why?

Abstract (arXiv)

We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don't. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation, or natural language. Experimental results show that our model effectively learns world dynamics, along with how to communicate them. It is able to correctly forecast "what happens next" given an English sentence over 80% of the time, outperforming a 100x larger, text-to-text approach by over 10%. Likewise, its natural language summaries of physical interactions are also judged by humans as more accurate than LM alternatives. We present comprehensive analysis showing room for future work.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 1 Jun 2021.

Builds on

27 of 44 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  11. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
15 more
  1. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  2. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  3. 2017Embodied Question Answeringcited by 335
  4. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  5. 2020Experience Grounds Languagecited by 241
  6. 2017IQA: Visual Question Answering in Interactive Environmentscited by 154
  7. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  8. 2019Fusion of Detected Objects in Text for Visual Question Answeringcited by 74
  9. 2015Actions ~ Transformationscited by 30
  10. 2017Zero-Shot Activity Recognition with Verb Attribute Inductioncited by 15
  11. 2019Effective Use of Transformer Networks for Entity Trackingcited by 11
  12. 2019ViCo: Word Embeddings from Visual Co-occurrencescited by 10
  13. 2019Extending Machine Language Models toward Human-Level Language Understandingcited by 7
  14. 2019Pre-Learning Environment Representations for Data-Efficient Neural Instruction Followingcited by 6
  15. 2018The Fast and the Flexible: training neural networks to learn to follow instructions from small datacited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 16; 2023: 16; 2024: 7; 2025: 3; 2026: 1.

47 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  5. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  6. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  7. 2021Skill Induction and Planning with Latent Languagecited by 99
  8. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  9. 2023Embodied Task Planning with Large Language Modelscited by 84
  10. 2021Word Acquisition in Neural Language Modelscited by 41
  11. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  12. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
35 more
  1. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
  2. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  3. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  4. 2023LLaMA Rider: Spurring Large Language Models to Explore the Open Worldcited by 21
  5. 2021Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it?cited by 18
  6. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  7. 2024OSCaR: Object State Captioning and State Change Representationcited by 16
  8. 2022DANLI: Deliberative Agent for Following Natural Language Instructionscited by 15
  9. 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
  10. 2022Behavior Cloned Transformers are Neurosymbolic Reasonerscited by 10
  11. 2024Neuro-Symbolic AI: Explainability, Challenges, and Future Trendscited by 10
  12. 2023Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Trackingcited by 8
  13. 2022EvEntS ReaLM: Event Reasoning of Entity States via Language Modelscited by 7
  14. 2022Language Modeling with Latent Situationscited by 6
  15. 2023Learning the meanings of function words from grounded language using a visual question answering modelcited by 6
  16. 2022Pretraining on Interactions for Learning Grounded Affordance Representationscited by 5
  17. 2024VCD: A Dataset for Visual Commonsense Discovery in Imagescited by 5
  18. 2022Learning Action Conditions from Instructional Manuals for Instruction Understandingcited by 4
  19. 2023Probing Physical Reasoning with Counter-Commonsense Contextcited by 4
  20. 2022Contextualized Sensorimotor Norms: multi-dimensional measures of sensorimotor strength for ambiguous English words, in contextcited by 3
  21. 2023Learning the Effects of Physical Actions in a Multi-modal Environmentcited by 3
  22. 2024Learning Planning Abstractions from Languagecited by 3
  23. 2025MINDSTORES: Memory-Informed Neural Decision Synthesis for Task-Oriented Reinforcement in Embodied Systemscited by 3
  24. 2022Learning Bidirectional Translation between Descriptions and Actions with Small Paired Datacited by 2
  25. 2023From Heuristic to Analytic: Cognitively Motivated Strategies for Coherent Physical Commonsense Reasoningcited by 2
  26. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  27. 2022A Theory of Natural Intelligencecited by 1
  28. 2022Trust in Language Grounding: a new AI challenge for human-robot teamscited by 1
  29. 2023A Data Source for Reasoning Embodied Agentscited by 1
  30. 2023Implicit Affordance Acquisition via Causal Action-Effect Modeling in the Video Domaincited by 1
  31. 2021Meaning Versus Information, Prediction Versus Memory, and Question Versus Answercited by 0
  32. 2022TRAC: A Textual Benchmark for Reasoning about Actions and Changecited by 0
  33. 2023Infusing Commonsense World Models with Graph Knowledgecited by 0
  34. 2024Bridging the Gap: Representation Spaces in Neuro-Symbolic AIcited by 0
  35. 2026Bridging Physical Reasoning and Task Generalization via Visual Action Outcome Reasoning Alignmentcited by 0