# A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping

arXiv: https://arxiv.org/abs/2409.19457 · HTML: https://mlsanity.com/paper/2409.19457/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2024.

## Built on by (most cited first, 6 of 6)

- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [GraspCoT: Integrating Physical Property Reasoning for 6-DoF Grasping under Flexible Language Instructions](https://mlsanity.com/paper/2503.16013/) (2025)
- [TALENT: Target-aware Efficient Tuning for Referring Image Segmentation](https://mlsanity.com/paper/2604.00609/) (2026)
- [Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoning](https://mlsanity.com/paper/2509.08126/) (2025)
- [LACY: A Vision-Language Model-based Language-Action Cycle for Self-Improving Robotic Manipulation](https://mlsanity.com/paper/2511.02239/) (2025)
- [Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation](https://mlsanity.com/paper/2606.30244/) (2026)

## Abstract (arXiv)

The language-guided robot grasping task requires a robot agent to integrate multimodal information from both visual and linguistic inputs to predict actions for target-driven grasping. While recent approaches utilizing Multimodal Large Language Models (MLLMs) have shown promising results, their extensive computation and data demands limit the feasibility of local deployment and customization. To address this, we propose a novel CLIP-based multimodal parameter-efficient tuning (PET) framework designed for three language-guided object grounding and grasping tasks: (1) Referring Expression Segmentation (RES), (2) Referring Grasp Synthesis (RGS), and (3) Referring Grasp Affordance (RGA). Our approach introduces two key innovations: a bi-directional vision-language adapter that aligns multimodal inputs for pixel-level language understanding and a depth fusion branch that incorporates geometric cues to facilitate robot grasping predictions. Experiment results demonstrate superior performance in the RES object grounding task compared with existing CLIP-based full-model tuning or PET approaches. In the RGS and RGA tasks, our model not only effectively interprets object attributes based on simple language descriptions but also shows strong potential for comprehending complex spatial reasoning scenarios, such as multiple identical objects present in the workspace. Project page: https://z.umn.edu/etog-etrg