# Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints

arXiv: https://arxiv.org/abs/2409.11518 · HTML: https://mlsanity.com/paper/2409.11518/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Affordances from Human Videos as a Versatile Representation for Robotics](https://mlsanity.com/paper/2304.08488/) (2023)
- [ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2312.16217/) (2023)
- [MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting](https://mlsanity.com/paper/2403.03174/) (2024)
- [PhraseCut: Language-based Image Segmentation in the Wild](https://mlsanity.com/paper/2008.01187/) (2020)
- [Highly Accurate Dichotomous Image Segmentation](https://mlsanity.com/paper/2203.03041/) (2022)
- [Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents](https://mlsanity.com/paper/2303.00855/) (2023)
- [AffordanceLLM: Grounding Affordance from Vision Language Models](https://mlsanity.com/paper/2401.06341/) (2024)
- [In the Eye of Transformer: Global-Local Correlation for Egocentric Gaze Estimation](https://mlsanity.com/paper/2208.04464/) (2022)
- [Unite-Divide-Unite: Joint Boosting Trunk and Structure for High-accuracy Dichotomous Image Segmentation](https://mlsanity.com/paper/2307.14052/) (2023)
- [OVAL-Prompt: Open-Vocabulary Affordance Localization for Robot Manipulation through LLM Affordance-Grounding](https://mlsanity.com/paper/2404.11000/) (2024)
- [Pay attention! - Robustifying a Deep Visuomotor Policy through Task-Focused Attention](https://mlsanity.com/paper/1809.10093/) (2018)
- [Learning Actions from Human Demonstration Video for Robotic Manipulation](https://mlsanity.com/paper/1909.04312/) (2019)
- [Text-driven Affordance Learning from Egocentric Vision](https://mlsanity.com/paper/2404.02523/) (2024)
- [ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals](https://mlsanity.com/paper/2309.12312/) (2023)
- [Contrastively Learning Visual Attention as Affordance Cues from Demonstrations for Robotic Grasping](https://mlsanity.com/paper/2104.00878/) (2021)
- [Generalizable task representation learning from human demonstration videos: a geometric approach](https://mlsanity.com/paper/2202.13604/) (2022)
- [Understanding Contexts Inside Robot and Human Manipulation Tasks through a Vision-Language Model and Ontology System in a Video Stream](https://mlsanity.com/paper/2003.01163/) (2020)
- [CLIPUNetr: Assisting Human-robot Interface for Uncalibrated Visual Servoing Control with CLIP-driven Referring Expression Segmentation](https://mlsanity.com/paper/2309.09183/) (2023)

## Built on by (most cited first, 5 of 5)

- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Latent Expression Generation for Referring Image Segmentation and Grounding](https://mlsanity.com/paper/2508.05123/) (2025)
- [PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding](https://mlsanity.com/paper/2505.20759/) (2025)
- [B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation](https://mlsanity.com/paper/2605.23500/) (2026)
- [Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker](https://mlsanity.com/paper/2605.25706/) (2026)

## Abstract (arXiv)

In this paper, we perform robot manipulation activities in real-world environments with language contexts by integrating a compact referring image segmentation model into the robot's perception module. First, we propose CLIPU2Net, a lightweight referring image segmentation model designed for fine-grain boundary and structure segmentation from language expressions. Then, we deploy the model in an eye-in-hand visual servoing system to enact robot control in the real world. The key to our system is the representation of salient visual information as geometric constraints, linking the robot's visual perception to actionable commands. Experimental results on 46 real-world robot manipulation tasks demonstrate that our method outperforms traditional visual servoing methods relying on labor-intensive feature annotations, excels in fine-grain referring image segmentation with a compact decoder size of 6.6 MB, and supports robot control across diverse contexts.