# Beyond Object Categories: Multi-Attribute Reference Understanding for Visual Grounding

arXiv: https://arxiv.org/abs/2503.19240 · HTML: https://mlsanity.com/paper/2503.19240/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 20 of 20)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [GRES: Generalized Referring Expression Segmentation](https://mlsanity.com/paper/2306.00968/) (2023)
- [SeqTR: A Simple yet Universal Network for Visual Grounding](https://mlsanity.com/paper/2203.16265/) (2022)
- [DetGPT: Detect What You Need via Reasoning](https://mlsanity.com/paper/2305.14167/) (2023)
- [Learning to Act Properly: Predicting and Explaining Affordances from Images](https://mlsanity.com/paper/1712.07576/) (2017)
- [DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection](https://mlsanity.com/paper/2404.09216/) (2024)
- [One-Shot Object Affordance Detection in the Wild](https://mlsanity.com/paper/2108.03658/) (2021)
- [YouRefIt: Embodied Reference Understanding with Language and Gesture](https://mlsanity.com/paper/2109.03413/) (2021)
- [One-Shot Affordance Detection](https://mlsanity.com/paper/2106.14747/) (2021)
- [TOIST: Task Oriented Instance Segmentation Transformer with Noun-Pronoun Distillation](https://mlsanity.com/paper/2210.10775/) (2022)
- [What Object Should I Use? - Task Driven Object Detection](https://mlsanity.com/paper/1904.03000/) (2019)
- [Understanding Embodied Reference with Touch-Line Transformer](https://mlsanity.com/paper/2210.05668/) (2022)
- [RIO: A Benchmark for Reasoning Intention-Oriented Objects in Open Environments](https://mlsanity.com/paper/2310.17290/) (2023)
- [Human-Robot Interaction in a Shared Augmented Reality Workspace](https://mlsanity.com/paper/2007.12656/) (2020)
- [Beyond Literal Descriptions: Understanding and Locating Open-World Objects Aligned with Human Intentions](https://mlsanity.com/paper/2402.11265/) (2024)

## Built on by (most cited first, 3 of 3)

- [Multi-label Instance-level Generalised Visual Grounding in Agriculture](https://mlsanity.com/paper/2603.06699/) (2026)
- [PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought](https://mlsanity.com/paper/2606.24539/) (2026)
- [VistaRef: Boosting Visual Spatial Orientation Awareness for Pointing-to-Object Detection](https://mlsanity.com/paper/2606.24498/) (2026)

## Abstract (arXiv)

Referring expression comprehension (REC) aims at achieving object localization based on natural language descriptions. However, existing REC approaches are constrained by object category descriptions and single-attribute intention descriptions, hindering their application in real-world scenarios. In natural human-robot interactions, users often express their desires through individual states and intentions, accompanied by guiding gestures, rather than detailed object descriptions. To address this challenge, we propose Multi-ref EC, a novel task framework that integrates state descriptions, derived intentions, and embodied gestures to locate target objects. We introduce the State-Intention-Gesture Attributes Reference (SIGAR) dataset, which combines state and intention expressions with embodied references. Through extensive experiments with various baseline models on SIGAR, we demonstrate that properly ordered multi-attribute references contribute to improved localization performance, revealing that single-attribute reference is insufficient for natural human-robot interaction scenarios. Our findings underscore the importance of multi-attribute reference expressions in advancing visual-language understanding.