mlsanityData to 14 Oct 2026

RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation

arXiv 2411.02704 ↗First posted 5 Nov 2024cs.RO cs.AI cs.CLcited by 43

Not on the map. Why?

Cited by 23 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We explore how intermediate policy representations can facilitate generalization by providing guidance on how to perform manipulation tasks. Existing representations such as language, goal images, and trajectory sketches have been shown to be helpful, but these representations either do not provide enough context or provide over-specified context that yields less robust policies. We propose conditioning policies on affordances, which capture the pose of the robot at key stages of the task. Affordances offer expressive yet lightweight abstractions, are easy for users to specify, and facilitate efficient learning by transferring knowledge from large internet datasets. Our method, RT-Affordance, is a hierarchical model that first proposes an affordance plan given the task language, and then conditions the policy on this affordance plan to perform manipulation. Our model can flexibly bridge heterogeneous sources of supervision including large web datasets and robot trajectories. We additionally train our model on cheap-to-collect in-domain affordance images, allowing us to learn new tasks without collecting any additional costly robot trajectories. We show on a diverse set of novel tasks how RT-Affordance exceeds the performance of existing methods by over 50%, and we empirically demonstrate that affordances are robust to novel settings. Videos available at https://snasiriany.me/rt-affordance

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 48 of its 54 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

48 of 54 references.

References · 48

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  6. 2023PaLM 2 Technical Reportcited by 1,162
  7. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  9. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  10. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
36 more
  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  2. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  3. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  4. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  5. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
  6. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  7. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  8. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  9. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  10. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 218
  11. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  12. 2024RT-H: Action Hierarchies Using Languagecited by 210
  13. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 208
  14. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  15. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  16. 2019Planning with Goal-Conditioned Policiescited by 149
  17. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  18. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 133
  19. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  20. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  21. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  22. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 111
  23. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  24. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  25. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 83
  26. 2020Transformers for One-Shot Visual Imitationcited by 78
  27. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  28. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  29. 2018Task-Embedded Control Networks for Few-Shot Imitation Learningcited by 66
  30. 2024Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformerscited by 52
  31. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 44
  32. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  33. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  34. 2020Affordances in Robotic Tasks -- A Surveycited by 28
  35. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  36. 2023Task-Oriented Grasp Prediction with Visual-Language Inputscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 24.

43 papers.

Sort:

By area:

  1. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 130
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  3. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
  4. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  5. 2025ChatVLA-2: Vision-Language-Action Model with Open-World Embodied Reasoning from Pretrained Knowledgecited by 49
  6. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  7. 2025CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learningcited by 40
  8. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  9. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordancecited by 31
  10. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 30
  11. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  12. 2024SeqAfford: Sequential 3D Affordance Reasoning via Multimodal Large Language Modelcited by 18
31 more
  1. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  2. 2026VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Modelscited by 17
  3. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Modelscited by 15
  4. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 13
  5. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  6. 2025Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understandingcited by 11
  7. 2026AwareVLN: Reasoning with Self-awareness for Vision-Language Navigationcited by 10
  8. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Modelscited by 9
  9. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignmentcited by 5
  10. 2025An Anatomy of Vision-Language-Action Models: From Modules to Milestones and Challengescited by 4
  11. 2026Affordance Agent Harness: Verification-Gated Skill Orchestrationcited by 4
  12. 2026Afford-VLA: Action-Aligned Visual Planning via Internalized Affordancecited by 4
  13. 2026DeVA: Decoupled Video-Action Model with physical guidance for robot policy learningcited by 4
  14. 2025O3Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulationcited by 3
  15. 2025Human Centric General Physical Intelligence for Agile Manufacturing Automationcited by 1
  16. 2026DAM-VLA: A Dynamic Action Model-Based Vision-Language-Action Framework for Robot Manipulationcited by 1
  17. 2026Panoramic Affordance Predictioncited by 1
  18. 2026TAP-VLA: Tactile Annotation Prompting for Vision Language Action Modelscited by 1
  19. 2026Cross-Embodiment Transfer via Behavior-Aligned Representationscited by 1
  20. 2026The Embodiment Gap in Robot Foundation Modelscited by 1
  21. 2025LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulationcited by 0
  22. 2026arg-VU: Affordance Reasoning with Physics-Aware 3D Geometry for Visual Understanding in Robotic Surgerycited by 0
  23. 2026AFUN: Towards an Affordance Foundation Model for Functionality Understandingcited by 0
  24. 20263DWay: Generalizing Robot Manipulation via 3D Consistent Waypointscited by 0
  25. 2026UniPart: Towards Zero-shot Language-Grounded 3D Part Segmentation for Embodied Interactioncited by 0
  26. 2026MaskHarness-WAM: Instance-Grounded Harnessing for Long-Horizon Robot Manipulationcited by 0
  27. 2026SCULPT-VLA: Learning Structured Control through Staged Action Groundingcited by 0
  28. 2026Gaze Prompts: Temporally Dense Human Attention for Vision-Language-Action Fine-Tuningcited by 0
  29. 2026BIND: Binding 3D Robot Actions to 2D Image Featurescited by 0
  30. 2026PAIR: Bridging Perception and Action in Vision-Language-Action Modelscited by 0
  31. 2026Phase-Conditioned Affordance Fields for SE(3)-Equivariant Visuomotor Policiescited by 0