# Grounding Synthetic Data Generation With Vision and Language Models

arXiv: https://arxiv.org/abs/2603.09625 · HTML: https://mlsanity.com/paper/2603.09625/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Self-improving MLLMs and synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 20 of 20)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Exploring Models and Data for Remote Sensing Image Caption Generation](https://mlsanity.com/paper/1712.07835/) (2017)
- [Pyramid Attention Network for Semantic Segmentation](https://mlsanity.com/paper/1805.10180/) (2018)
- [A U-Net Based Discriminator for Generative Adversarial Networks](https://mlsanity.com/paper/2002.12655/) (2020)
- [Distilling Diffusion Models into Conditional GANs](https://mlsanity.com/paper/2405.05967/) (2024)
- [SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?](https://mlsanity.com/paper/2402.01832/) (2024)
- [The GAN is dead; long live the GAN! A Modern GAN Baseline](https://mlsanity.com/paper/2501.05441/) (2025)
- [Generative AI for Synthetic Data Generation: Methods, Challenges and the Future](https://mlsanity.com/paper/2403.04190/) (2024)
- [Not All Samples Should Be Utilized Equally: Towards Understanding and Improving Dataset Distillation](https://mlsanity.com/paper/2408.12483/) (2024)
- [Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications](https://mlsanity.com/paper/2510.21887/) (2025)
- [Learning from Synthetic Data for Visual Grounding](https://mlsanity.com/paper/2403.13804/) (2024)

## Built on by (most cited first, 3 of 3)

- [LALE: Lightweight-Transformer Architecture for Land-Cover Estimation](https://mlsanity.com/paper/2606.02092/) (2026)
- [BELDE: Building a Large-scale Earth-observation Land-cover Dataset for Europe](https://mlsanity.com/paper/2606.20909/) (2026)
- [Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation](https://mlsanity.com/paper/2606.25128/) (2026)

## Abstract (arXiv)

Deep learning models benefit from increasing data diversity and volume, motivating synthetic data augmentation to improve existing datasets. However, existing evaluation metrics for synthetic data typically calculate latent feature similarity, which is difficult to interpret and does not always correlate with the contribution to downstream tasks. We propose a vision-language grounded framework for interpretable synthetic data augmentation and evaluation in remote sensing. Our approach combines generative models, semantic segmentation and image captioning with vision and language models. Based on this framework, we introduce ARAS400k: A large-scale Remote sensing dataset Augmented with Synthetic data for segmentation and captioning, containing 100k real images and 300k synthetic images, each paired with segmentation maps and descriptions. ARAS400k enables the automated evaluation of synthetic data by analyzing semantic composition, minimizing caption redundancy, and verifying cross-modal consistency between visual structures and language descriptions. Experimental results indicate that while models trained exclusively on synthetic data reach competitive performance levels, those trained with augmented data (a combination of real and synthetic images) consistently outperform real-data baselines. Consequently, this work establishes a scalable benchmark for remote sensing tasks, specifically in semantic segmentation and image captioning. The dataset is available at zenodo.org/records/18890661 and the code base at github.com/caglarmert/ARAS400k.