mlsanityData to 7 Oct 2026

Experience Grounds Language

arXiv 2004.10151 ↗First posted 21 Apr 2020cs.CL cs.AI cs.LGcited by 241

Not on the map. Why?

Abstract (arXiv)

Language understanding research is held back by a failure to relate language to the physical world it describes and to the social interactions it facilitates. Despite the incredible effectiveness of language processing models to tackle tasks after being trained on text alone, successful linguistic communication relies on a shared experience of the world. It is this shared experience that makes utterances meaningful. Natural language processing is a diverse field, and progress throughout its development has come from new representational theories, modeling techniques, data collection paradigms, and tasks. We posit that the present success of representation learning approaches trained on large, text-only corpora requires the parallel tradition of research on the broader physical and social context of language to address the deeper questions of communication.

Key facts. Cited by 241 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 53; 2022: 53; 2023: 57; 2024: 24; 2025: 29; 2026: 25.

241 papers.

Sort:

By area:

  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  3. 2023Language Models Represent Space and Timecited by 306
  4. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  5. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  6. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  7. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  8. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  9. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 162
  10. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  11. 2022Language Models as Agent Modelscited by 131
  12. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
38 more
  1. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  2. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  3. 2021WebQA: Multihop and Multimodal QAcited by 109
  4. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  5. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  6. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
  7. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  8. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  9. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 78
  10. 2022Meaning without reference in large language modelscited by 76
  11. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  12. 2022A fine-grained comparison of pragmatic language understanding in humans and language modelscited by 69
  13. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  14. 2023Learning to Model the World with Languagecited by 66
  15. 2024Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Modelscited by 65
  16. 2022A New Path: Scaling Vision-and-Language Navigation with Synthetic Instructions and Imitation Learningcited by 60
  17. 2023Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Modelscited by 59
  18. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 57
  19. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  20. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  21. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  22. 2021PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D Worldcited by 47
  23. 2022Retrospectives on the Embodied AI Workshopcited by 46
  24. 2021Grounding 'Grounding' in NLPcited by 44
  25. 2022Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionalitycited by 44
  26. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  27. 2021Language Grounding with 3D Objectscited by 43
  28. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  29. 2022Neural Approaches to Conversational Information Retrievalcited by 41
  30. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  31. 20243D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucinationcited by 36
  32. 2022Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Followingcited by 33
  33. 2021The World of an Octopus: How Reporting Bias Influences a Language Model's Perception of Colorcited by 32
  34. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  35. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  36. 2022Underspecification in Scene Description-to-Depiction Taskscited by 30
  37. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  38. 2023Beyond Surface Statistics: Scene Representations in a Latent Diffusion Modelcited by 29

50 most cited of 241 All 241 (JSON)