# MixGen: A New Multi-Modal Data Augmentation

arXiv: https://arxiv.org/abs/2206.08358 · HTML: https://mlsanity.com/paper/2206.08358/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 72 references that are in the map.
- First posted on arXiv 16 Jun 2022.

## Builds on (most cited first, 30 of 56)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)

## Built on by (most cited first, 30 of 61)

- [RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://mlsanity.com/paper/2502.21257/) (2025)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [Teaching Structured Vision&Language Concepts to Vision&Language Models](https://mlsanity.com/paper/2211.11733/) (2022)
- [DocFormerv2: Local Features for Document Understanding](https://mlsanity.com/paper/2306.01733/) (2023)
- [A Survey on Data Synthesis and Augmentation for Large Language Models](https://mlsanity.com/paper/2410.12896/) (2024)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Measuring Progress in Fine-grained Vision-and-Language Understanding](https://mlsanity.com/paper/2305.07558/) (2023)
- [Unifying Vision-Language Representation Space with Single-tower Transformer](https://mlsanity.com/paper/2211.11153/) (2022)
- [A Survey on Mixup Augmentations and Beyond](https://mlsanity.com/paper/2409.05202/) (2024)
- [Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment](https://mlsanity.com/paper/2208.13628/) (2022)
- [Learning Multimodal Data Augmentation in Feature Space](https://mlsanity.com/paper/2212.14453/) (2022)
- [Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification](https://mlsanity.com/paper/2403.01944/) (2024)
- [YORO -- Lightweight End to End Visual Grounding](https://mlsanity.com/paper/2211.07912/) (2022)
- [Cross-modality Data Augmentation for End-to-End Sign Language Translation](https://mlsanity.com/paper/2305.11096/) (2023)
- [Object Detection with Multimodal Large Vision-Language Models: An In-depth Review](https://mlsanity.com/paper/2508.19294/) (2025)
- [Enhancing Vision-Language Pre-training with Rich Supervisions](https://mlsanity.com/paper/2403.03346/) (2024)
- [Exploring Train and Test-Time Augmentations for Audio-Language Learning](https://mlsanity.com/paper/2210.17143/) (2022)
- [Data-Efficient Multimodal Fusion on a Single GPU](https://mlsanity.com/paper/2312.10144/) (2023)
- [MAP: A Model-agnostic Pretraining Framework for Click-through Rate Prediction](https://mlsanity.com/paper/2308.01737/) (2023)
- [Multimodal Data Augmentation for Visual-Infrared Person ReID with Corrupted Data](https://mlsanity.com/paper/2211.11925/) (2022)
- [A Survey of Data Synthesis Approaches](https://mlsanity.com/paper/2407.03672/) (2024)
- [Bridging Domain Generalization to Multimodal Domain Generalization via Unified Representations](https://mlsanity.com/paper/2507.03304/) (2025)
- [Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learning](https://mlsanity.com/paper/2305.16379/) (2023)
- [Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptation](https://mlsanity.com/paper/2404.13425/) (2024)
- [Synergistic Prompting for Robust Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2507.07802/) (2025)
- [3VL: Using Trees to Improve Vision-Language Models' Interpretability](https://mlsanity.com/paper/2312.17345/) (2023)
- [Finding NeMo: Negative-mined Mosaic Augmentation for Referring Image Segmentation](https://mlsanity.com/paper/2411.01494/) (2024)
- [Improved Automatic Diabetic Retinopathy Severity Classification Using Deep Multimodal Fusion of UWF-CFP and OCTA Images](https://mlsanity.com/paper/2310.01912/) (2023)
- [Does VLM Classification Benefit from LLM Description Semantics?](https://mlsanity.com/paper/2412.11917/) (2024)
- [Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learning](https://mlsanity.com/paper/2211.01796/) (2022)

## Abstract (arXiv)

Data augmentation is a necessity to enhance data efficiency in deep learning. For vision-language pre-training, data is only augmented either for images or for text in previous works. In this paper, we present MixGen: a joint data augmentation for vision-language representation learning to further improve data efficiency. It generates new image-text pairs with semantic relationships preserved by interpolating images and concatenating text. It's simple, and can be plug-and-played into existing pipelines. We evaluate MixGen on four architectures, including CLIP, ViLT, ALBEF and TCL, across five downstream vision-language tasks to show its versatility and effectiveness. For example, adding MixGen in ALBEF pre-training leads to absolute performance improvements on downstream tasks: image-text retrieval (+6.2% on COCO fine-tuned and +5.3% on Flicker30K zero-shot), visual grounding (+0.9% on RefCOCO+), visual reasoning (+0.9% on NLVR2), visual question answering (+0.3% on VQA2.0), and visual entailment (+0.4% on SNLI-VE).