# OCID-Ref: A 3D Robotic Dataset with Embodied Language for Clutter Scene Grounding

arXiv: https://arxiv.org/abs/2103.07679 · HTML: https://mlsanity.com/paper/2103.07679/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 13 Mar 2021.

## Builds on (most cited first, 11 of 11)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Dynamic Graph Attention for Referring Expression Comprehension](https://mlsanity.com/paper/1909.08164/) (2019)
- [EasyLabel: A Semi-Automatic Pixel-wise Object Annotation Tool for Creating Robotic RGB-D Datasets](https://mlsanity.com/paper/1902.01626/) (2019)
- [Graph-Structured Referring Expression Reasoning in The Wild](https://mlsanity.com/paper/2004.08814/) (2020)
- [Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction](https://mlsanity.com/paper/1806.03831/) (2018)
- [Cops-Ref: A new Dataset and Task on Compositional Referring Expression Comprehension](https://mlsanity.com/paper/2003.00403/) (2020)
- [GDN: A Coarse-To-Fine (C2F) Representation for End-To-End 6-DoF Grasp Detection](https://mlsanity.com/paper/2010.10695/) (2020)

## Built on by (most cited first, 16 of 16)

- [Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models](https://mlsanity.com/paper/2402.07865/) (2024)
- [Language-Driven Representation Learning for Robotics](https://mlsanity.com/paper/2302.12766/) (2023)
- [ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension](https://mlsanity.com/paper/2204.05991/) (2022)
- [Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration](https://mlsanity.com/paper/2412.13180/) (2024)
- [Learning Manipulation by Predicting Interaction](https://mlsanity.com/paper/2406.00439/) (2024)
- [SUGAR: Pre-training 3D Visual Representations for Robotics](https://mlsanity.com/paper/2404.01491/) (2024)
- [Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Clutter](https://mlsanity.com/paper/2311.05779/) (2023)
- [Locality Alignment Improves Vision-Language Models](https://mlsanity.com/paper/2410.11087/) (2024)
- [Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control](https://mlsanity.com/paper/2405.05852/) (2024)
- [Mind to Hand: Purposeful Robotic Control via Embodied Reasoning](https://mlsanity.com/paper/2512.08580/) (2025)
- [7B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancement](https://mlsanity.com/paper/2412.06845/) (2024)
- [Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLA](https://mlsanity.com/paper/2512.22208/) (2025)
- [GVCCI: Lifelong Learning of Visual Grounding for Language-Guided Robotic Manipulation](https://mlsanity.com/paper/2307.05963/) (2023)
- [Should VLMs be Pre-trained with Image Data?](https://mlsanity.com/paper/2503.07603/) (2025)
- [Do VLMs Need Vision Transformers? Evaluating State Space Models as Vision Encoders](https://mlsanity.com/paper/2603.19209/) (2026)
- [Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?](https://mlsanity.com/paper/2610.18834/) (2026)

## Abstract (arXiv)

To effectively apply robots in working environments and assist humans, it is essential to develop and evaluate how visual grounding (VG) can affect machine performance on occluded objects. However, current VG works are limited in working environments, such as offices and warehouses, where objects are usually occluded due to space utilization issues. In our work, we propose a novel OCID-Ref dataset featuring a referring expression segmentation task with referring expressions of occluded objects. OCID-Ref consists of 305,694 referring expressions from 2,300 scenes with providing RGB image and point cloud inputs. To resolve challenging occlusion issues, we argue that it's crucial to take advantage of both 2D and 3D signals to resolve challenging occlusion issues. Our experimental results demonstrate the effectiveness of aggregating 2D and 3D signals but referring to occluded objects still remains challenging for the modern visual grounding systems. OCID-Ref is publicly available at https://github.com/lluma/OCID-Ref