# FG-CLIP: Fine-Grained Visual and Textual Alignment

arXiv: https://arxiv.org/abs/2505.05071 · HTML: https://mlsanity.com/paper/2505.05071/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 68 references that are in the map.
- First posted on arXiv 8 May 2025.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)

## Built on by (most cited first, 30 of 81)

- [Emu3.5: Native Multimodal Models are World Learners](https://mlsanity.com/paper/2510.26583/) (2025)
- [MemVerse: Multimodal Memory for Lifelong Learning Agents](https://mlsanity.com/paper/2512.03627/) (2025)
- [RzenEmbed: Towards Comprehensive Multimodal Retrieval](https://mlsanity.com/paper/2510.27350/) (2025)
- [UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset](https://mlsanity.com/paper/2510.20661/) (2025)
- [Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models](https://mlsanity.com/paper/2509.13031/) (2025)
- [FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Model](https://mlsanity.com/paper/2510.10921/) (2025)
- [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026)
- [LMM-Det: Make Large Multimodal Models Excel in Object Detection](https://mlsanity.com/paper/2507.18300/) (2025)
- [L2P: Unlocking Latent Potential for Pixel Generation](https://mlsanity.com/paper/2605.12013/) (2026)
- [VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions](https://mlsanity.com/paper/2508.02329/) (2025)
- [More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment](https://mlsanity.com/paper/2508.06036/) (2025)
- [POMA-3D: The Point Map Way to 3D Scene Understanding](https://mlsanity.com/paper/2511.16567/) (2025)
- [SuperCLIP: CLIP with Simple Classification Supervision](https://mlsanity.com/paper/2512.14480/) (2025)
- [MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models](https://mlsanity.com/paper/2601.20433/) (2026)
- [Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignment](https://mlsanity.com/paper/2603.17655/) (2026)
- [A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering](https://mlsanity.com/paper/2510.04428/) (2025)
- [BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Models](https://mlsanity.com/paper/2510.20095/) (2025)
- [Attention Grounded Enhancement for Visual Document Retrieval](https://mlsanity.com/paper/2511.13415/) (2025)
- [EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligence](https://mlsanity.com/paper/2512.15160/) (2025)
- [Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplification](https://mlsanity.com/paper/2602.05729/) (2026)
- [SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognition](https://mlsanity.com/paper/2603.17729/) (2026)
- [SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation](https://mlsanity.com/paper/2509.03897/) (2025)
- [HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment](https://mlsanity.com/paper/2511.06653/) (2025)
- [Boosting Medical Visual Understanding From Multi-Granular Language Learning](https://mlsanity.com/paper/2511.15943/) (2025)
- [ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matching](https://mlsanity.com/paper/2512.17178/) (2025)
- [BindCLIP: A Unified Contrastive-Generative Representation Learning Framework for Virtual Screening](https://mlsanity.com/paper/2602.15236/) (2026)
- [GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learning](https://mlsanity.com/paper/2603.09566/) (2026)
- [Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2603.22042/) (2026)
- [Mitigating Multimodal Hallucination via Phase-wise Self-reward](https://mlsanity.com/paper/2604.17982/) (2026)
- [T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalability](https://mlsanity.com/paper/2604.18573/) (2026)

## Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but struggles with fine-grained understanding due to its focus on coarse-grained short captions. To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances fine-grained understanding through three key innovations. First, we leverage large multimodal models to generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, a high-quality dataset is constructed with 12 million images and 40 million region-specific bounding boxes aligned with detailed captions to ensure precise, context-rich representations. Third, 10 million hard fine-grained negative samples are incorporated to improve the model's ability to distinguish subtle semantic differences. We construct a comprehensive dataset, termed FineHARD, by integrating high-quality region-specific annotations with hard fine-grained negative samples. Corresponding training methods are meticulously designed for these data. Extensive experiments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-art methods across various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-grained image details and improving overall model performance. The data, code, and models are available at https://github.com/360CVGroup/FG-CLIP.