mlsanityData to 7 Oct 2026

Grounding Language with Visual Affordances over Unstructured Data

arXiv 2210.01911 ↗First posted 4 Oct 2022cs.RO cs.AI cs.CLcited by 101

Not on the map. Why?

Cited by 31 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent works have shown that Large Language Models (LLMs) can be applied to ground natural language to a wide variety of robot skills. However, in practice, learning multi-task, language-conditioned robotic skills typically requires large-scale data collection and frequent human intervention to reset the environment or help correcting the current policies. In this work, we propose a novel approach to efficiently learn general-purpose language-conditioned robot skills from unstructured, offline and reset-free data in the real world by exploiting a self-supervised visuo-lingual affordance model, which requires annotating as little as 1% of the total data with language. We evaluate our method in extensive experiments both in simulated and real-world robotic tasks, achieving state-of-the-art performance on the challenging CALVIN benchmark and learning over 25 distinct visuomotor manipulation tasks with a single policy in the real world. We find that when paired with LLMs to break down abstract natural language instructions into subgoals via few-shot prompting, our method is capable of completing long-horizon, multi-tier tasks in the real world, while requiring an order of magnitude less data than previous approaches. Code and videos are available at http://hulc2.cs.uni-freiburg.de

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

35 of 39 references.

References · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2022A Generalist Agentcited by 802
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
23 more
  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  2. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  3. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  4. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  5. 2019Learning Latent Plans from Playcited by 310
  6. 2020Experience Grounds Languagecited by 241
  7. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  8. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  9. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  10. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 124
  11. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  12. 2018Mapping Instructions to Actions in 3D Environments with Visual Goal Predictioncited by 101
  13. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  14. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  15. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  16. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  17. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  18. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  19. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  20. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 27
  21. 2020Robot Object Retrieval with Contextual Natural Language Queriescited by 26
  22. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  23. 2020Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Followingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 21; 2024: 37; 2025: 27; 2026: 14.

101 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  5. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  7. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  8. 2022Visual Language Maps for Robot Navigationcited by 362
  9. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  10. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  11. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  12. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
38 more
  1. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  2. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  5. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  6. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  7. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  8. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  9. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  10. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  11. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  12. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  13. 2023Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planningcited by 88
  14. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  15. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  16. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  17. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
  18. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  19. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  20. 2024Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videoscited by 65
  21. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  22. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  23. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  24. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 58
  25. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  26. 2023One-Shot Open Affordance Learning with Foundation Modelscited by 45
  27. 2023Large Language Models Are Semi-Parametric Reinforcement Learning Agentscited by 42
  28. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  29. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  30. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  31. 2024Learning Manipulation by Predicting Interactioncited by 36
  32. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  33. 2023Prompt, Plan, Perform: LLM-based Humanoid Control via Quantized Imitation Learningcited by 31
  34. 2023Audio Visual Language Maps for Robot Navigationcited by 30
  35. 2023Human-in-the-Loop Task and Motion Planning for Imitation Learningcited by 28
  36. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  37. 2025MLA: A Multisensory Language-Action Model for Multimodal Understanding and Forecasting in Robotic Manipulationcited by 22
  38. 2025WristWorld: Generating Wrist-Views via 4D World Models for Robotic Manipulationcited by 21

50 most cited of 101 All 101 (JSON)