# ReSeDis: A Dataset for Referring-based Object Search across Large-Scale Image Collections

arXiv: https://arxiv.org/abs/2506.15180 · HTML: https://mlsanity.com/paper/2506.15180/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 18 Jun 2025.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [A Fast and Accurate One-Stage Approach to Visual Grounding](https://mlsanity.com/paper/1908.06354/) (2019)
- [Learning to Assemble Neural Module Tree Networks for Visual Grounding](https://mlsanity.com/paper/1812.03299/) (2018)
- [Improving One-stage Visual Grounding by Recursive Sub-query Construction](https://mlsanity.com/paper/2008.01059/) (2020)
- [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [Natural Language Object Retrieval](https://mlsanity.com/paper/1511.04164/) (2015)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension](https://mlsanity.com/paper/1909.07072/) (2019)
- [Dynamic Graph Attention for Referring Expression Comprehension](https://mlsanity.com/paper/1909.08164/) (2019)
- [Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation](https://mlsanity.com/paper/2105.01839/) (2021)
- [Learning to Compose and Reason with Language Tree Structures for Visual Grounding](https://mlsanity.com/paper/1906.01784/) (2019)
- [Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing](https://mlsanity.com/paper/1903.00839/) (2019)
- [Dynamic Multimodal Instance Segmentation guided by natural language queries](https://mlsanity.com/paper/1807.02257/) (2018)
- [Grounding Referring Expressions in Images by Variational Context](https://mlsanity.com/paper/1712.01892/) (2017)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)

## Built on by (most cited first, 1 of 1)

- [KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge](https://mlsanity.com/paper/2508.14080/) (2025)

## Abstract (arXiv)

Large-scale visual search engines are expected to solve a dual problem at once: (i) locate every image that truly contains the object described by a sentence and (ii) identify the object's bounding box or exact pixels within each hit. Existing techniques address only one side of this challenge. Visual grounding yields tight boxes and masks but rests on the unrealistic assumption that the object is present in every test image, producing a flood of false alarms when applied to web-scale collections. Text-to-image retrieval excels at sifting through massive databases to rank relevant images, yet it stops at whole-image matches and offers no fine-grained localization. We introduce Referring Search and Discovery (ReSeDis), the first task that unifies corpus-level retrieval with pixel-level grounding. Given a free-form description, a ReSeDis model must decide whether the queried object appears in each image and, if so, where it is, returning bounding boxes or segmentation masks. To enable rigorous study, we curate a benchmark in which every description maps uniquely to object instances scattered across a large, diverse corpus, eliminating unintended matches. We further design a task-specific metric that jointly scores retrieval recall and localization precision. Finally, we provide a straightforward zero-shot baseline using a frozen vision-language model, revealing significant headroom for future study. ReSeDis offers a realistic, end-to-end testbed for building the next generation of robust and scalable multimodal search systems.