# Enhancing Fine-Grained Vision-Language Pretraining with Negative Augmented Samples

arXiv: https://arxiv.org/abs/2412.10029 · HTML: https://mlsanity.com/paper/2412.10029/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 12 of its 71 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data](https://mlsanity.com/paper/2001.07966/) (2020)
- [Products-10K: A Large-scale Product Recognition Dataset](https://mlsanity.com/paper/2008.10545/) (2020)
- [Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding](https://mlsanity.com/paper/2306.08832/) (2023)
- [UNIMO-2: End-to-End Unified Vision-Language Grounded Learning](https://mlsanity.com/paper/2203.09067/) (2022)

## Built on by (most cited first, 7 of 7)

- [A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarks](https://mlsanity.com/paper/2506.08227/) (2025)
- [VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions](https://mlsanity.com/paper/2508.02329/) (2025)
- [CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models](https://mlsanity.com/paper/2412.16869/) (2024)
- [VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models](https://mlsanity.com/paper/2509.14571/) (2025)
- [A New Method to Capturing Compositional Knowledge in Linguistic Space](https://mlsanity.com/paper/2412.15632/) (2024)
- [Towards Highly Transferable Vision-Language Attack via Semantic-Augmented Dynamic Contrastive Interaction](https://mlsanity.com/paper/2603.04839/) (2026)
- [GRASP: Granularity-Aware Region Alignment and Semantic Prototype Learning for Fine-Grained Cross-Modal Understanding in Drone Views](https://mlsanity.com/paper/2608.09270/) (2026)

## Abstract (arXiv)

Existing Vision-Language Pretraining (VLP) methods have achieved remarkable improvements across a variety of vision-language tasks, confirming their effectiveness in capturing coarse-grained semantic correlations. However, their capability for fine-grained understanding, which is critical for many nuanced vision-language applications, remains limited. Prevailing VLP models often overlook the intricate distinctions in expressing different modal features and typically depend on the similarity of holistic features for cross-modal interactions. Moreover, these models directly align and integrate features from different modalities, focusing more on coarse-grained general representations, thus failing to capture the nuanced differences necessary for tasks demanding a more detailed perception. In response to these limitations, we introduce Negative Augmented Samples(NAS), a refined vision-language pretraining model that innovatively incorporates NAS to specifically address the challenge of fine-grained understanding. NAS utilizes a Visual Dictionary(VD) as a semantic bridge between visual and linguistic domains. Additionally, it employs a Negative Visual Augmentation(NVA) method based on the VD to generate challenging negative image samples. These samples deviate from positive samples exclusively at the token level, thereby necessitating that the model discerns the subtle disparities between positive and negative samples with greater precision. Comprehensive experiments validate the efficacy of NAS components and underscore its potential to enhance fine-grained vision-language comprehension.