# Semantic-aware Data Augmentation for Text-to-image Synthesis

arXiv: https://arxiv.org/abs/2312.07951 · HTML: https://mlsanity.com/paper/2312.07951/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 38 references that are in the map.
- First posted on arXiv 13 Dec 2023.

## Builds on (most cited first, 7 of 7)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP](https://mlsanity.com/paper/2203.00386/) (2022)
- [Survey: Image Mixing and Deleting for Data Augmentation](https://mlsanity.com/paper/2106.07085/) (2021)

## Built on by (most cited first, 4 of 4)

- [SCMix: Stochastic Compound Mixing for Open Compound Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/2405.14278/) (2024)
- [Revisiting Mutual Information Maximization for Generalized Category Discovery](https://mlsanity.com/paper/2405.20711/) (2024)
- [LoFT: LoRA-fused Training Dataset Generation with Few-shot Guidance](https://mlsanity.com/paper/2505.11703/) (2025)
- [Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching](https://mlsanity.com/paper/2507.09256/) (2025)

## Abstract (arXiv)

Data augmentation has been recently leveraged as an effective regularizer in various vision-language deep neural networks. However, in text-to-image synthesis (T2Isyn), current augmentation wisdom still suffers from the semantic mismatch between augmented paired data. Even worse, semantic collapse may occur when generated images are less semantically constrained. In this paper, we develop a novel Semantic-aware Data Augmentation (SADA) framework dedicated to T2Isyn. In particular, we propose to augment texts in the semantic space via an Implicit Textual Semantic Preserving Augmentation (ITA), in conjunction with a specifically designed Image Semantic Regularization Loss (Lr) as Generated Image Semantic Conservation, to cope well with semantic mismatch and collapse. As one major contribution, we theoretically show that ITA can certify better text-image consistency while Lr regularizing the semantics of generated images would avoid semantic collapse and enhance image quality. Extensive experiments validate that SADA enhances text-image consistency and improves image quality significantly in T2Isyn models across various backbones. Especially, incorporating SADA during the tuning process of Stable Diffusion models also yields performance improvements.