# Harlequin: Color-driven Generation of Synthetic Data for Referring Expression Comprehension

arXiv: https://arxiv.org/abs/2411.14807 · HTML: https://mlsanity.com/paper/2411.14807/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 5 of its 45 references that are in the map.
- First posted on arXiv 22 Nov 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 5 of 5)

- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Parents and Children: Distinguishing Multimodal DeepFakes from Natural Images](https://mlsanity.com/paper/2304.00500/) (2023)
- [Dataset of Industrial Metal Objects](https://mlsanity.com/paper/2208.04052/) (2022)

## Built on by (most cited first, 3 of 3)

- [7Bench: a Comprehensive Benchmark for Layout-guided Text-to-image Models](https://mlsanity.com/paper/2508.12919/) (2025)
- [Making Dialogue Grounding Data Rich: A Three-Tier Data Synthesis Framework for Generalized Referring Expression Comprehension](https://mlsanity.com/paper/2512.02791/) (2025)
- [Benchmarking Layout-Guided Diffusion Models through Unified Semantic-Spatial Evaluation in Closed and Open Settings](https://mlsanity.com/paper/2604.25358/) (2026)

## Abstract (arXiv)

Referring Expression Comprehension (REC) aims to identify a particular object in a scene by a natural language expression, and is an important topic in visual language understanding. State-of-the-art methods for this task are based on deep learning, which generally requires expensive and manually labeled annotations. Some works tackle the problem with limited-supervision learning or relying on Large Vision and Language Models. However, the development of techniques to synthesize labeled data is overlooked. In this paper, we propose a novel framework that generates artificial data for the REC task, taking into account both textual and visual modalities. At first, our pipeline processes existing data to create variations in the annotations. Then, it generates an image using altered annotations as guidance. The result of this pipeline is a new dataset, called Harlequin, made by more than 1M queries. This approach eliminates manual data collection and annotation, enabling scalability and facilitating arbitrary complexity. We pre-train three REC models on Harlequin, then fine-tuned and evaluated on human-annotated datasets. Our experiments show that the pre-training on artificial data is beneficial for performance.