# RoboGround: Robotic Manipulation with Grounded Vision-Language Priors

arXiv: https://arxiv.org/abs/2504.21530 · HTML: https://mlsanity.com/paper/2504.21530/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 50 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks](https://mlsanity.com/paper/2311.06242/) (2023)
- [RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots](https://mlsanity.com/paper/2406.02523/) (2024)
- [Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation](https://mlsanity.com/paper/2312.13139/) (2023)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://mlsanity.com/paper/2310.10639/) (2023)
- [Any-point Trajectory Modeling for Policy Learning](https://mlsanity.com/paper/2401.00025/) (2023)
- [RT-H: Action Hierarchies Using Language](https://mlsanity.com/paper/2403.01823/) (2024)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)

## Built on by (most cited first, 30 of 51)

- [Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications](https://mlsanity.com/paper/2510.07077/) (2025)
- [InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policy](https://mlsanity.com/paper/2510.13778/) (2025)
- [InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation](https://mlsanity.com/paper/2507.17520/) (2025)
- [InternVLA-A1: Unifying Understanding, Generation and Action for Robotic Manipulation](https://mlsanity.com/paper/2601.02456/) (2026)
- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding](https://mlsanity.com/paper/2506.01853/) (2025)
- [VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models](https://mlsanity.com/paper/2603.22003/) (2026)
- [FutureVLA: Joint Visuomotor Prediction for Vision-Language-Action Model](https://mlsanity.com/paper/2603.10712/) (2026)
- [OAT: Ordered Action Tokenization](https://mlsanity.com/paper/2602.04215/) (2026)
- [Point What You Mean: Visually Grounded Instruction Policy](https://mlsanity.com/paper/2512.18933/) (2025)
- [Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Grounding](https://mlsanity.com/paper/2512.22519/) (2025)
- [ST4VLA: Spatially Guided Training for Vision-Language-Action Models](https://mlsanity.com/paper/2602.10109/) (2026)
- [HiVLA: A Visual-Grounded-Centric Hierarchical Embodied Manipulation System](https://mlsanity.com/paper/2604.14125/) (2026)
- [Gaze2Act: Gaze-Conditioned Vision-Language-Action Policies for Interactive Robot Manipulation](https://mlsanity.com/paper/2605.30282/) (2026)
- [Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting](https://mlsanity.com/paper/2512.20014/) (2025)
- [IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks](https://mlsanity.com/paper/2509.24768/) (2025)
- [Afford-VLA: Action-Aligned Visual Planning via Internalized Affordance](https://mlsanity.com/paper/2605.24203/) (2026)
- [Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulation](https://mlsanity.com/paper/2510.07975/) (2025)
- [MultihopSpatial: Multi-hop Compositional Spatial Reasoning Benchmark for Vision-Language Model](https://mlsanity.com/paper/2603.18892/) (2026)
- [Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Intervention](https://mlsanity.com/paper/2511.22555/) (2025)
- [From Instruction to Event: Sound-Triggered Mobile Manipulation](https://mlsanity.com/paper/2601.21667/) (2026)
- [AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement](https://mlsanity.com/paper/2604.10432/) (2026)
- [From Imagined Futures to Executable Actions: Mixture of Latent Actions for Robot Manipulation](https://mlsanity.com/paper/2605.12167/) (2026)
- [From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding](https://mlsanity.com/paper/2605.15951/) (2026)
- [Direct Action-Head Injection of A Grounded 3D Point Unlocks Spatial and Task Generalization](https://mlsanity.com/paper/2606.27663/) (2026)
- [Hijacking Robots with a Piece of Paper: A Systematic Study of Physical Prompt Injection in VLM-Controlled Robots](https://mlsanity.com/paper/2608.05715/) (2026)
- [InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation](https://mlsanity.com/paper/2608.22990/) (2026)
- [AR2-4FV: Anchored Referring and Re-identification for Long-Term Grounding in Fixed-View Videos](https://mlsanity.com/paper/2603.07758/) (2026)
- [Decoupling Semantics and Geometric Grounding: Spatial Visual Prompts for Language-Conditioned Imitation Learning](https://mlsanity.com/paper/2606.25360/) (2026)
- [SAM3D-Guided Object-Centric Representation Alignment for Vision-Language-Action Models](https://mlsanity.com/paper/2607.25912/) (2026)

## Abstract (arXiv)

Recent advancements in robotic manipulation have highlighted the potential of intermediate representations for improving policy generalization. In this work, we explore grounding masks as an effective intermediate representation, balancing two key advantages: (1) effective spatial guidance that specifies target objects and placement areas while also conveying information about object shape and size, and (2) broad generalization potential driven by large-scale vision-language models pretrained on diverse grounding datasets. We introduce RoboGround, a grounding-aware robotic manipulation system that leverages grounding masks as an intermediate representation to guide policy networks in object manipulation tasks. To further explore and enhance generalization, we propose an automated pipeline for generating large-scale, simulated data with a diverse set of objects and instructions. Extensive experiments show the value of our dataset and the effectiveness of grounding masks as intermediate guidance, significantly enhancing the generalization abilities of robot policies.