# OmniDexVLG: Learning Dexterous Grasp Generation from Vision Language Model-Guided Grasp Semantics, Taxonomy and Functional Affordance arXiv: https://arxiv.org/abs/2512.03874 ยท HTML: https://mlsanity.com/paper/2512.03874/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. - Builds on 31 of its 37 references that are in the map. - First posted on arXiv 3 Dec 2025. ## Placed in - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 30 of 31) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [PointLLM: Empowering Large Language Models to Understand Point Clouds](https://mlsanity.com/paper/2308.16911/) (2023) - [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022) - [RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://mlsanity.com/paper/2502.21257/) (2025) - [General In-Hand Object Rotation with Vision and Touch](https://mlsanity.com/paper/2309.09979/) (2023) - [Synthesizing Diverse and Physically Stable Grasps with Arbitrary Hand Structures using Differentiable Force Closure Estimator](https://mlsanity.com/paper/2104.09194/) (2021) - [DexDiffuser: Generating Dexterous Grasps with Diffusion Models](https://mlsanity.com/paper/2402.02989/) (2024) - [DextrAH-G: Pixels-to-Action Dexterous Arm-Hand Grasping with Geometric Fabrics](https://mlsanity.com/paper/2407.02274/) (2024) - [GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping](https://mlsanity.com/paper/2307.13204/) (2023) - [Grasp as You Say: Language-guided Dexterous Grasp Generation](https://mlsanity.com/paper/2405.19291/) (2024) - [RobotGPT: Robot Manipulation Learning from ChatGPT](https://mlsanity.com/paper/2312.01421/) (2023) - [Dexterous Functional Grasping](https://mlsanity.com/paper/2312.02975/) (2023) - [GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training](https://mlsanity.com/paper/2507.13097/) (2025) - [PyRoki: A Modular Toolkit for Robot Kinematic Optimization](https://mlsanity.com/paper/2505.03728/) (2025) - [SemGrasp: Semantic Grasp Generation via Language Aligned Discretization](https://mlsanity.com/paper/2404.03590/) (2024) - [ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models](https://mlsanity.com/paper/2505.16517/) (2025) - [DexGrasp Anything: Towards Universal Robotic Dexterous Grasping with Physics Awareness](https://mlsanity.com/paper/2503.08257/) (2025) - [Dexonomy: Synthesizing All Dexterous Grasp Types in a Grasp Taxonomy](https://mlsanity.com/paper/2504.18829/) (2025) - [What Foundation Models can Bring for Robot Learning in Manipulation : A Survey](https://mlsanity.com/paper/2404.18201/) (2024) - [Learning 6-DoF Fine-grained Grasp Detection Based on Part Affordance Grounding](https://mlsanity.com/paper/2301.11564/) (2023) - [FoundationGrasp: Generalizable Task-Oriented Grasping with Foundation Models](https://mlsanity.com/paper/2404.10399/) (2024) - [AnyDexGrasp: General Dexterous Grasping for Different Hands with Human-level Learning Efficiency](https://mlsanity.com/paper/2502.16420/) (2025) - [Multi-GraspLLM: A Multimodal LLM for Multi-Hand Semantic Guided Grasp Generation](https://mlsanity.com/paper/2412.08468/) (2024) - [ContactDexNet: Multi-fingered Robotic Hand Grasping in Cluttered Environments through Hand-object Contact Semantic Mapping](https://mlsanity.com/paper/2404.08844/) (2024) - [Cross-Category Functional Grasp Transfer](https://mlsanity.com/paper/2405.08310/) (2024) - [Don't Let Your Robot be Harmful: Responsible Robotic Manipulation via Safety-as-Policy](https://mlsanity.com/paper/2411.18289/) (2024) - [GrainGrasp: Dexterous Grasp Generation with Fine-grained Contact Guidance](https://mlsanity.com/paper/2405.09310/) (2024) - [ToolEENet: Tool Affordance 6D Pose Estimation](https://mlsanity.com/paper/2404.04193/) (2024) - [IFG: Internet-Scale Guidance for Functional Grasping Generation](https://mlsanity.com/paper/2511.09558/) (2025) ## Built on by (most cited first, 2 of 2) - [Adaptive Vision-Language Grasping via Composable Foundation Priors and Generalizable Grasp Synthesis](https://mlsanity.com/paper/2609.04096/) (2026) - [CoToGrasp: Contact-Topology-Conditioned Dexterous Grasp Synthesis via Canonical Workspace Learning](https://mlsanity.com/paper/2608.19776/) (2026) ## Abstract (arXiv) Dexterous grasp generation aims to produce grasp poses that align with task requirements and human interpretable grasp semantics. However, achieving semantically controllable dexterous grasp synthesis remains highly challenging due to the lack of unified modeling of multiple semantic dimensions, including grasp taxonomy, contact semantics, and functional affordance. To address these limitations, we present OmniDexVLG, a multimodal, semantics aware grasp generation framework capable of producing structurally diverse and semantically coherent dexterous grasps under joint language and visual guidance. Our approach begins with OmniDexDataGen, a semantic rich dexterous grasp dataset generation pipeline that integrates grasp taxonomy guided configuration sampling, functional affordance contact point sampling, taxonomy aware differential force closure grasp sampling, and physics based optimization and validation, enabling systematic coverage of diverse grasp types. We further introduce OmniDexReasoner, a multimodal grasp type semantic reasoning module that leverages multi agent collaboration, retrieval augmented generation, and chain of thought reasoning to infer grasp related semantics and generate high quality annotations that align language instructions with task specific grasp intent. Building upon these components, we develop a unified Vision Language Grasping generation model that explicitly incorporates grasp taxonomy, contact structure, and functional affordance semantics, enabling fine grained control over grasp synthesis from natural language instructions. Extensive experiments in simulation and real world object grasping and ablation studies demonstrate that our method substantially outperforms state of the art approaches in terms of grasp diversity, contact semantic diversity, functional affordance diversity, and semantic consistency.