mlsanityData to 14 Oct 2026

A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter

arXiv 2302.12610 ↗First posted 24 Feb 2023cs.ROcited by 34

On the map: Vision-language-action models → 3D and spatial-aware VLA

Abstract (arXiv)

We focus on the task of language-conditioned grasping in clutter, in which a robot is supposed to grasp the target object based on a language instruction. Previous works separately conduct visual grounding to localize the target object, and generate a grasp for that object. However, these works require object labels or visual attributes for grounding, which calls for handcrafted rules in planner and restricts the range of language instructions. In this paper, we propose to jointly model vision, language and action with object-centric representation. Our method is applicable under more flexible language instructions, and not limited by visual grounding error. Besides, by utilizing the powerful priors from the pre-trained multi-modal model and grasp model, sample efficiency is effectively improved and the sim2real problem is relived without additional data for transfer. A series of experiments carried out in simulation and real world indicate that our method can achieve better task success rate by less times of motion under more flexible language instructions. Moreover, our method is capable of generalizing better to scenarios with unseen objects and language instructions. Our code is available at https://github.com/xukechun/Vision-Language-Grasping

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in 3D and spatial-aware VLA, one of 13 directions in Vision-language-action models. Builds on 35 of its 49 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

35 of 49 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  10. 2016Modeling Context in Referring Expressionscited by 1,063
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
23 more
  1. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  2. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 218
  3. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  4. 2019Soft Actor-Critic for Discrete Action Settingscited by 174
  5. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  6. 2017Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matchingcited by 125
  7. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  8. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  9. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 62
  10. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 56
  11. 2017Multi-Task Domain Adaptation for Deep Learning of Instance Grasping from Simulationcited by 36
  12. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  13. 2021OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulationcited by 24
  14. 2021A Joint Network for Grasp Detection Conditioned on Natural Language Commandscited by 23
  15. 2021Semantically Grounded Object Matching for Robust Robotic Scene Rearrangementcited by 21
  16. 2017End-to-End Learning of Semantic Graspingcited by 20
  17. 2019A Deep Learning Approach to Grasping the Invisiblecited by 19
  18. 2022Interactive Robotic Grasping with Attribute-Guided Disambiguationcited by 19
  19. 2020Visuomotor Mechanical Search: Learning to Retrieve Target Objects in Cluttercited by 16
  20. 2021Visual Foresight Trees for Object Retrieval from Clutter with Nonprehensile Rearrangementcited by 15
  21. 2021Efficient learning of goal-oriented push-grasping synergy in cluttercited by 12
  22. 2021GATER: Learning Grasp-Action-Target Embeddings and Relations for Task-Specific Graspingcited by 4
  23. 2022Grasping as Inference: Reactive Grasping in Heavily Cluttered Environmentcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 16; 2025: 9; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 35
  2. 2023Grasp-Anything: Large-scale Grasp Dataset from Foundation Modelscited by 33
  3. 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queriescited by 27
  4. 2024Language-driven Grasp Detectioncited by 26
  5. 2024Reasoning Grasping via Multimodal Large Language Modelcited by 20
  6. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 18
  7. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  8. 2023Applications of Large Scale Foundation Models for Autonomous Drivingcited by 14
  9. 2025GLOVER++: Unleashing the Potential of Affordance Learning from Human Behaviors for Robotic Manipulationcited by 14
  10. 2024Lightweight Language-driven Grasp Detection using Conditional Consistency Modelcited by 9
  11. 2025GarmentPile: Point-Level Visual Affordance Guided Retrieval and Adaptation for Cluttered Garments Manipulationcited by 9
  12. 2025Free-form language-based robotic reasoning and graspingcited by 8
22 more
  1. 2024HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Modelscited by 7
  2. 2024RT-Grasp: Reasoning Tuning Robotic Grasping via Multi-modal Large Language Modelcited by 7
  3. 2024Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Priorcited by 5
  4. 2024Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applicationscited by 5
  5. 2023Object-centric Inference for Language Conditioned Placement: A Foundation Model based Approachcited by 4
  6. 2024GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learningcited by 3
  7. 2025Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Cluttercited by 3
  8. 2024Language-driven Grasp Detection with Mask-guided Attentioncited by 2
  9. 2024LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusioncited by 2
  10. 2024Target-Oriented Object Grasping via Multimodal Human Guidancecited by 2
  11. 2024Advances in Transformers for Robotic Applications: A Reviewcited by 2
  12. 2025VCoT-Grasp: Grasp Foundation Models with Visual Chain-of-Thought Reasoning for Language-driven Grasp Generationcited by 2
  13. 2024Learning Dual-Arm Push and Grasp Synergy in Dense Cluttercited by 1
  14. 2025Language-Guided Grasp Detection with Coarse-to-Fine Learning for Robotic Manipulationcited by 1
  15. 2026GarmentPile++: Affordance-Driven Cluttered Garments Retrieval with Vision-Language Reasoningcited by 1
  16. 2026RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulationcited by 1
  17. 2026GraspFoM: Towards Reconstruction-Driven Robotic Grasping with 3D Foundation Priorscited by 1
  18. 2023QwenGrasp: A Usage of Large Vision-Language Model for Target-Oriented Graspingcited by 0
  19. 2024Constrained Robotic Navigation on Preferred Terrains Using LLMs and Speech Instruction: Exploiting the Power of Adverbscited by 0
  20. 2025Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoningcited by 0
  21. 2025VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Modelscited by 0
  22. 2026Learning Object-Centric Spatial Reasoning for Sequential Manipulation in Cluttered Environmentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping
  2. 2025Improving Generalization of Language-Conditioned Robot Manipulation
  3. 2024Towards Open-World Grasping with Large Vision-Language Models
  4. 2026Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning
  5. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding
  6. 2026Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
  7. 2026Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization
  8. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping