mlsanityData to 7 Oct 2026

RAAP: Retrieval-Augmented Affordance Prediction with Cross-Image Action Alignment

arXiv 2603.29419 ↗First posted 31 Mar 2026cs.RO cs.AI cs.CVcited by 0

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Understanding object affordances is essential for enabling robots to perform purposeful and fine-grained interactions in diverse and unstructured environments. However, existing approaches either rely on retrieval, which is fragile due to sparsity and coverage gaps, or on large-scale models, which frequently mislocalize contact points and mispredict post-contact actions when applied to unseen categories, thereby hindering robust generalization. We introduce Retrieval-Augmented Affordance Prediction (RAAP), a framework that unifies affordance retrieval with alignment-based learning. By decoupling static contact localization and dynamic action direction, RAAP transfers contact points via dense correspondence and predicts action directions through a retrieval-augmented alignment model that consolidates multiple references with dual-weighted attention. Trained on compact subsets of DROID and HOI4D with as few as tens of samples per task, RAAP achieves consistent performance across unseen objects and categories, and enables zero-shot robotic manipulation in both simulation and the real world. Project website: https://github.com/SEU-VIPGroup/RAAP.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 43 of its 57 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  8. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  9. 2023A Survey on Multimodal Large Language Modelscited by 898
  10. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  11. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  12. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
31 more
  1. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  2. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  3. 2016Deep Image Retrieval: Learning global representations for image searchcited by 147
  4. 2021Fine-Grained Image Analysis with Deep Learning: A Surveycited by 135
  5. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  6. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  7. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  8. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  9. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  10. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  11. 2020Learning Dexterous Grasping with Object-Centric Visual Affordancescited by 80
  12. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  13. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  14. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 78
  15. 2024Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulationcited by 69
  16. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 56
  17. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  18. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 54
  19. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  20. 2024Graspness Discovery in Clutters for Fast and Accurate Grasp Detectioncited by 47
  21. 2025A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulationcited by 44
  22. 2024Retrieval Augmented Time Series Forecastingcited by 40
  23. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  24. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 35
  25. 2024DenseMatcher: Learning 3D Semantic Correspondence for Category-Level Manipulation from a Single Democited by 34
  26. 2022One-Shot Transfer of Affordance Regions? AffCorrs!cited by 31
  27. 2024Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representationscited by 23
  28. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  29. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluationcited by 17
  30. 2023Attribute-Aware Deep Hashing with Self-Consistency for Large-Scale Fine-Grained Image Retrievalcited by 8
  31. 2024Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalizationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation
  2. 2025Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
  3. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment
  4. 2026AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
  5. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
  6. 2025RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
  7. 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
  8. 2026Affordance2Action: Task-Conditioned Scene-level Affordance Grounding for Real-Time Manipulation ↗