mlsanityData to 7 Oct 2026

Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents

arXiv 2303.00855 ↗First posted 1 Mar 2023cs.RO cs.AI cs.CLcited by 66

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Recent progress in large language models (LLMs) has demonstrated the ability to learn and leverage Internet-scale knowledge through pre-training with autoregressive models. Unfortunately, applying such models to settings with embodied agents, such as robots, is challenging due to their lack of experience with the physical world, inability to parse non-language observations, and ignorance of rewards or safety constraints that robots may require. On the other hand, language-conditioned robotic policies that learn from interaction data can provide the necessary grounding that allows the agent to be correctly situated in the real world, but such policies are limited by the lack of high-level semantic understanding due to the limited breadth of the interaction data available for training them. Thus, if we want to make use of the semantic knowledge in a language model while still situating it in an embodied setting, we must construct an action sequence that is both likely according to the language model and also realizable according to grounded models of the environment. We frame this as a problem similar to probabilistic filtering: decode a sequence that both has high probability under the language model and high probability under a set of grounded model objectives. We demonstrate how such grounded models can be obtained across three simulation and real-world domains, and that the proposed decoding strategy is able to solve complex, long-horizon embodiment tasks in a robotic setting by leveraging the knowledge of both models. The project's website can be found at grounded-decoding.github.io.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 73 of its 97 references that are in the map. First posted on arXiv 1 Mar 2023.

Builds on

73 of 97 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models8 Dec
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents18 Jan

Other references · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
38 more
  1. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  7. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  8. 2022A Generalist Agentcited by 802
  9. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  10. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  11. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  12. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  13. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  14. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  15. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  16. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  17. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  18. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  19. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  20. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  21. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  22. 2022Visual Language Maps for Robot Navigationcited by 362
  23. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  24. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  25. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  26. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  27. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  28. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  29. 2018Semi-parametric Topological Memory for Navigationcited by 203
  30. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  31. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  32. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  33. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  34. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  35. 2023Translating Natural Language to Planning Goals with Large-Language Modelscited by 132
  36. 2018Learning to Write with Cooperative Discriminatorscited by 119
  37. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  38. 2020If beam search is the answer, what was the question?cited by 114

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 22; 2025: 28; 2026: 10.

66 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  3. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  4. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  5. 2023Large Language Models for Robotics: A Surveycited by 132
  6. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  7. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  8. 2023SayCanPay: Heuristic Planning with Large Language Models using Learnable Domain Knowledgecited by 49
  9. 2024Bidirectional Decoding: Improving Action Chunking via Guided Test-Time Samplingcited by 46
  10. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  11. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  12. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
38 more
  1. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  2. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  3. 2023Interactive Natural Language Processingcited by 31
  4. 2024ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoningcited by 27
  5. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  6. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  7. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 20
  8. 2023AlphaBlock: Embodied Finetuning for Vision-Language Reasoning in Robot Manipulationcited by 19
  9. 2024Trust the PRoC3S: Solving Long-Horizon Robotics Problems with LLMs and Constraint Satisfactioncited by 19
  10. 2025NeSyC: A Neuro-symbolic Continual Learner For Complex Embodied Tasks In Open Domainscited by 15
  11. 2024Easier Said Than Done: Unpacking Intent-Behavior Gap in Jailbreaking LLM-Based Robotscited by 14
  12. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  13. 2024Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Accesscited by 9
  14. 2024E2CL: Exploration-based Error Correction Learning for Embodied Agentscited by 9
  15. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraintscited by 9
  16. 2024GraphAgent: Agentic Graph Language Assistantcited by 8
  17. 2025REMAC: Self-Reflective and Self-Evolving Multi-Agent Collaboration for Long-Horizon Robot Manipulationcited by 8
  18. 2024VeriGraph: Scene Graphs for Execution Verifiable Robot Planningcited by 7
  19. 2025SPIRAL: Symbolic LLM Planning via Grounded and Reflective Searchcited by 7
  20. 2024Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraintscited by 5
  21. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  22. 2025AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robotcited by 5
  23. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  24. 2024WorldAPIs: The World Is Worth How Many APIs? A Thought Experimentcited by 4
  25. 2024Skills Made to Order: Efficient Acquisition of Robot Cooking Skills Guided by Multiple Forms of Internet Datacited by 4
  26. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  27. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 4
  28. 2025Learn as Individuals, Evolve as a Team: Multi-agent LLMs Adaptation in Embodied Environmentscited by 4
  29. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  30. 2024Foundation Models to the Rescue: Deadlock Resolution in Connected Multi-Robot Systemscited by 3
  31. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  32. 2024Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skillscited by 2
  33. 2025Large Language Models as Common-Sense Heuristicscited by 2
  34. 2025RoboInspector: Unveiling the Unreliability of Policy Code for LLM-enabled Robotic Manipulationcited by 2
  35. 2025Vision Language Models Cannot Plan, but Can They Formalize?cited by 2
  36. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmarkcited by 2
  37. 2025Efficient Navigation in Unknown Indoor Environments with Vision-Language Modelscited by 2
  38. 2024REVECA: Adaptive Planning and Trajectory-based Validation in Cooperative Language Agents using Information Relevance and Relative Proximitycited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Grounding Large Language Models In Embodied Environment With Imperfect World Models
  2. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents
  3. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents
  4. 2026ACCORD: Action-Conditioned Contextual Grounding for Language Agents
  5. 2026RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
  6. 2025Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Models
  7. 2025From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems
  8. 2024Grounding Multimodal Large Language Models in Actions