mlsanityData to 7 Oct 2026

HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models

arXiv 2409.10419 ↗First posted 16 Sep 2024cs.RO cs.AIcited by 7

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Robots interacting with humans through natural language can unlock numerous applications such as Referring Grasp Synthesis (RGS). Given a text query, RGS determines a stable grasp pose to manipulate the referred object in the robot's workspace. RGS comprises two steps: visual grounding and grasp pose estimation. Recent studies leverage powerful Vision-Language Models (VLMs) for visually grounding free-flowing natural language in real-world robotic execution. However, comparisons in complex, cluttered environments with multiple instances of the same object are lacking. This paper introduces HiFi-CS, featuring hierarchical application of Featurewise Linear Modulation (FiLM) to fuse image and text embeddings, enhancing visual grounding for complex attribute rich text queries encountered in robotic grasping. Visual grounding associates an object in 2D/3D space with natural language input and is studied in two scenarios: Closed and Open Vocabulary. HiFi-CS features a lightweight decoder combined with a frozen VLM and outperforms competitive baselines in closed vocabulary settings while being 100x smaller in size. Our model can effectively guide open-set object detectors like GroundedSAM to enhance open-vocabulary performance. We validate our approach through real-world RGS experiments using a 7-DOF robotic arm, achieving 90.33% visual grounding accuracy in 15 tabletop scenes. Our codebase is provided here: https://github.com/vineet2104/hifics

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 85 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

66 of 85 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Reasoning Grasping via Multimodal Large Language Model9 Feb
  2. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration20 Nov
  3. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  6. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  7. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Segment Anythingcited by 8,158
  3. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
38 more
  1. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  2. 2021Image Segmentation Using Text and Image Promptscited by 509
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  4. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  5. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  6. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  7. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  8. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  9. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  10. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250
  11. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 205
  12. 2023Universal Instance Perception as Object Discovery and Retrievalcited by 190
  13. 2017Modulating early visual processing by languagecited by 167
  14. 2023PolyFormer: Referring Image Segmentation as Sequential Polygon Generationcited by 147
  15. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  16. 2015Natural Language Object Retrievalcited by 124
  17. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 115
  18. 2021Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentationcited by 111
  19. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  20. 2021Cross-Modal Progressive Comprehension for Referring Segmentationcited by 102
  21. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  22. 2018Jacquard: A Large Scale Dataset for Robotic Grasp Detectioncited by 86
  23. 2021TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Groundingcited by 86
  24. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 84
  25. 2022Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoningcited by 69
  26. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  27. 2021End-to-end Trainable Deep Neural Network for Robotic Grasp Detection and Semantic Segmentation from RGBcited by 44
  28. 2022Text2Pos: Text-to-Point-Cloud Cross-Modal Localizationcited by 40
  29. 2023VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenescited by 40
  30. 2024Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4Vcited by 40
  31. 2021RGB Matters: Learning 7-DoF Grasp Poses on Monocular RGBD Imagescited by 38
  32. 2023Referring Image Segmentation Using Text Supervisioncited by 35
  33. 2021Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Imagescited by 33
  34. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  35. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  36. 2020GSNet: Joint Vehicle Pose and Shape Reconstruction with Geometrical and Scene-aware Supervisioncited by 28
  37. 2024Language-driven Grasp Detectioncited by 26
  38. 2024Grounding LLMs For Robot Task Planning Using Closed-loop State Feedbackcited by 25

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  2. 2024Toward Visual Grounding: A Surveycited by 52
  3. 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Cluttercited by 20
  4. 2024GraspMamba: A Mamba-based Language-driven Grasp Detection Framework with Hierarchical Feature Learningcited by 3
  5. 2025Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edgecited by 0
  6. 2026Training-free Suction Grasp Detection for Deformed Aseptic Cartons Using Vision-Language Models and Geometric Surface Scoringcited by 0
  7. 2026RefGuard: Identity-Aware Language-Guided Robot Manipulation via Joint Target-Anchor-Frame Groundingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulation
  2. 2024A Parameter-Efficient Tuning Framework for Language-guided Object Grounding and Robot Grasping
  3. 2025Attribute-based Object Grounding and Robot Grasp Detection with Spatial Reasoning
  4. 2024Towards Open-World Grasping with Large Vision-Language Models
  5. 2025MapleGrasp: Mask-guided Feature Pooling for Language-driven Efficient Robotic Grasping
  6. 2025Free-form language-based robotic reasoning and grasping
  7. 2025Improving Generalization of Language-Conditioned Robot Manipulation
  8. 2025Training-free Task-oriented Grasp Generation