# GMAIL: Generative Modality Alignment for generated Image Learning

arXiv: https://arxiv.org/abs/2602.15368 · HTML: https://mlsanity.com/paper/2602.15368/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 17 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2301.08243/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort](https://mlsanity.com/paper/2104.06490/) (2021)
- [Learning Vision from Models Rivals Learning Vision from Data](https://mlsanity.com/paper/2312.17742/) (2023)
- [Task2Sim : Towards Effective Pre-training and Transfer from Synthetic Data](https://mlsanity.com/paper/2112.00054/) (2021)
- [SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesis](https://mlsanity.com/paper/2311.03355/) (2023)
- [IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning](https://mlsanity.com/paper/2409.18046/) (2024)

## Built on by (most cited first, 1 of 1)

- [LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation](https://mlsanity.com/paper/2603.27693/) (2026)

## Abstract (arXiv)

Generative models have made it possible to synthesize highly realistic images, potentially providing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as a separate modality from real images. Instead of indiscriminately replacing real images with generated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross-modality alignment loss and then employ this aligned model to further train various vision-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework can be easily incorporated with various vision-language models, and we demonstrate its efficacy throughout extensive experiments. For example, our framework significantly improves performance on image captioning, zero-shot image retrieval, zero-shot image classification, and long caption retrieval tasks. It also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.