Contrastive Visual Data Augmentation
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Large multimodal models (LMMs) often struggle to recognize novel concepts, as they rely on pre-trained knowledge and have limited ability to capture subtle visual details. Domain-specific knowledge gaps in training also make them prone to confusing visually similar, commonly misrepresented, or low-resource concepts. To help LMMs better align nuanced visual features with language, improving their ability to recognize and reason about novel or rare concepts, we propose a Contrastive visual Data Augmentation (CoDA) strategy. CoDA extracts key contrastive textual and visual features of target concepts against the known concepts they are misrecognized as, and then uses multimodal generative models to produce targeted synthetic data. Automatic filtering of extracted features and augmented images is implemented to guarantee their quality, as verified by human annotators. We show the effectiveness and efficiency of CoDA on low-resource concept and diverse scene recognition datasets including INaturalist and SUN. We additionally collect NovelSpecies, a benchmark dataset consisting of newly discovered animal species that are guaranteed to be unseen by LMMs. LLaVA-1.6 1-shot updating results on these three datasets show CoDA significantly improves SOTA visual data augmentation strategies by 12.3% (NovelSpecies), 5.1% (SUN), and 6.0% (iNat) absolute gains in accuracy.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 47 references that are in the map. First posted on arXiv 24 Feb 2025.
Builds on
35 of 47 references.
Closest ancestors · 2
Same direction, newest first
- 2024Rethinking VLMs and LLMs for Image Classification3 Oct
- 2024Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings12 Mar
Other references · 33
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Visual Instruction Tuningcited by 7,869
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2024GPT-4o System Cardcited by 4,588
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
- 2016Matching Networks for One Shot Learningcited by 2,255
- 2018On First-Order Meta-Learning Algorithmscited by 1,018
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
21 more
- 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
- 2020Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?cited by 437
- 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
- 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
- 2019A Baseline for Few-Shot Image Classificationcited by 298
- 2020SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularizationcited by 149
- 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
- 2022Image Data Augmentation for Deep Learning: A Surveycited by 96
- 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
- 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
- 2022Matching Feature Sets for Few-Shot Image Classificationcited by 25
- 2023Image Data Augmentation Approaches: A Comprehensive Survey and Future directionscited by 23
- 2019STaDA: Style Transfer as Data Augmentationcited by 21
- 2022Learning Multimodal Data Augmentation in Feature Spacecited by 18
- 2023Sample-Efficient Learning of Novel Visual Conceptscited by 8
- 2023Localizing Active Objects from Egocentric Vision with Symbolic World Knowledgecited by 8
- 2023Towards reporting bias in visual-language datasets: bimodal augmentation by decoupling object-attribute associationcited by 7
- 2021Cut-Thumbnail: A Novel Data Augmentation for Convolutional Neural Networkcited by 5
- 2021StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Texturescited by 3
- 2024ARMADA: Attribute-Based Multimodal Data Augmentationcited by 2
- 2024Verbalized Representation Learning for Interpretable Few-Shot Generalizationcited by 2
Built on by
Citing papers per year: 2025: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples
- 2024COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training
- 2026Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models
- 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models
- 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
- 2024Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models
- 2024AmCLR: Unified Augmented Learning for Cross-Modal Representations
- 2024Exploring the Distinctiveness and Fidelity of the Descriptions Generated by Large Vision-Language Models