mlsanityData to 7 Oct 2026

UniAff: A Unified Representation of Affordances for Tool Usage and Articulation with Vision-Language Models

arXiv 2409.20551 ↗First posted 30 Sep 2024cs.ROcited by 6

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Previous studies on robotic manipulation are based on a limited understanding of the underlying 3D motion constraints and affordances. To address these challenges, we propose a comprehensive paradigm, termed UniAff, that integrates 3D object-centric manipulation and task understanding in a unified formulation. Specifically, we constructed a dataset labeled with manipulation-related key attributes, comprising 900 articulated objects from 19 categories and 600 tools from 12 categories. Furthermore, we leverage MLLMs to infer object-centric representations for manipulation tasks, including affordance recognition and reasoning about 3D motion constraints. Comprehensive experiments in both simulation and real-world settings indicate that UniAff significantly improves the generalization of robotic manipulation for tools and articulated objects. We hope that UniAff will serve as a general baseline for unified robotic manipulation tasks in the future. Images, videos, dataset, and code are published on the project website at:https://sites.google.com/view/uni-aff/home

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 45 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

38 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulation17 Jun
  2. 2024A3VLM: Actionable Articulation-Aware Vision Language Model11 Jun
  3. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models17 Mar
  4. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec
  5. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Segment Anythingcited by 8,158
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  9. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
21 more
  1. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  2. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  3. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  4. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  5. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  6. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 135
  7. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 130
  8. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  9. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  10. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  11. 2023Bridging Zero-shot Object Navigation and Foundation Models through Pixel-Guided Navigation Skillcited by 75
  12. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  13. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  14. 2024Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Trackingcited by 48
  15. 2023HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose Annotations, Affordances, and Reconstructionscited by 47
  16. 2024Learning Precise Affordances from Egocentric Videos for Robotic Manipulationcited by 34
  17. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
  18. 2024RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Modelcited by 28
  19. 2023Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMscited by 19
  20. 2023LGMCTS: Language-Guided Monte-Carlo Tree Search for Executable Semantic Object Rearrangementcited by 11
  21. 2023Understanding URDF: A Dataset and Analysiscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
  2. 2025RAGNet: Large-scale Reasoning-based Affordance Segmentation Benchmark towards General Graspingcited by 11
  3. 2025Adversarial Data Collection: Human-Collaborative Perturbations for Efficient and Robust Robotic Imitation Learningcited by 9
  4. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  5. 2026Creative Robot Tool Use by Counterfactual Reasoningcited by 0
  6. 2026AFUN: Towards an Affordance Foundation Model for Functionality Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
  2. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
  3. 2025A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation
  4. 2026VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances
  5. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  6. 2026AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation
  7. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
  8. 2025RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation