mlsanityData to 14 Oct 2026

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

arXiv 2204.01691 ↗First posted 4 Apr 2022cs.RO cs.CL cs.LGcited by 2,348

Not on the map. Why?

Cited by 516 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.

Key facts. Cited by 2,348 papers in the map (data through 14 Oct 2026). Builds on 63 of its 103 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

63 of 103 references.

References · 63

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  12. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
38 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  2. 2015Prioritized Experience Replaycited by 1,099
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  6. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  10. 2018Universal Sentence Encodercited by 489
  11. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  12. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  13. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
  14. 2018Semi-parametric Topological Memory for Navigationcited by 209
  15. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  16. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  17. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  18. 2021Episodic Transformer for Vision-and-Language Navigationcited by 156
  19. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  20. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  21. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  22. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  23. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  24. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  25. 2021Skill Induction and Planning with Latent Languagecited by 99
  26. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 98
  27. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  28. 2022A data-driven approach for learning to control computerscited by 88
  29. 2019Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generationcited by 83
  30. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 81
  31. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  32. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 76
  33. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  34. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 70
  35. 2018Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstrationcited by 61
  36. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 61
  37. 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
  38. 2021Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completioncited by 58

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 93; 2023: 451; 2024: 554; 2025: 542; 2026: 708.

2,348 papers.

Sort:

By area:

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2023A Survey of Large Language Modelscited by 2,576
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  8. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  9. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
38 more
  1. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,761
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  9. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  10. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  12. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  13. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  14. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  15. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  16. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  17. 2022A Generalist Agentcited by 808
  18. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  19. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  20. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  21. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  22. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  23. 2022PAL: Program-aided Language Modelscited by 652
  24. 2023A Comprehensive Overview of Large Language Modelscited by 648
  25. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  26. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 598
  27. 2023Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Modelscited by 574
  28. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  29. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  30. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  31. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  32. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  33. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  34. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  35. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  36. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 501
  37. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  38. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478

50 most cited of 2,348 All 2,348 (JSON)