mlsanityData to 7 Oct 2026

FG-CLIP: Fine-Grained Visual and Textual Alignment

arXiv 2505.05071 ↗First posted 8 May 2025cs.CV cs.AIcited by 78

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) excels in multimodal tasks such as image-text retrieval and zero-shot classification but struggles with fine-grained understanding due to its focus on coarse-grained short captions. To address this, we propose Fine-Grained CLIP (FG-CLIP), which enhances fine-grained understanding through three key innovations. First, we leverage large multimodal models to generate 1.6 billion long caption-image pairs for capturing global-level semantic details. Second, a high-quality dataset is constructed with 12 million images and 40 million region-specific bounding boxes aligned with detailed captions to ensure precise, context-rich representations. Third, 10 million hard fine-grained negative samples are incorporated to improve the model's ability to distinguish subtle semantic differences. We construct a comprehensive dataset, termed FineHARD, by integrating high-quality region-specific annotations with hard fine-grained negative samples. Corresponding training methods are meticulously designed for these data. Extensive experiments demonstrate that FG-CLIP outperforms the original CLIP and other state-of-the-art methods across various downstream tasks, including fine-grained understanding, open-vocabulary object detection, image-text retrieval, and general multimodal benchmarks. These results highlight FG-CLIP's effectiveness in capturing fine-grained image details and improving overall model performance. The data, code, and models are available at https://github.com/360CVGroup/FG-CLIP.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 56 of its 68 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

56 of 68 references.

References · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2017Mask R-CNNcited by 7,259
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  13. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  14. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  15. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  16. 2021Extract Free Dense Labels from CLIPcited by 580
  17. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  18. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  19. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 498
  20. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  21. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  22. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  23. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  24. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 214
  25. 2023PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documentscited by 212
  26. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  27. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  28. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  29. 2023CLIPSelf: Vision Transformer Distills Itself for Open-Vocabulary Dense Predictioncited by 114
  30. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  31. 2022Learning Object-Language Alignments for Open-Vocabulary Object Detectioncited by 95
  32. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  33. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  34. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  35. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenescited by 55
  36. 2018Memory Attention Networks for Skeleton-based Action Recognitioncited by 36
  37. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  38. 2024Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinementcited by 30

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 48.

78 papers.

Sort:

By area:

  1. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  2. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 31
  3. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 28
  4. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Modelscited by 16
  5. 2025UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Datasetcited by 16
  6. 2025FG-CLIP 2: A Bilingual Fine-grained Vision-Language Alignment Modelcited by 11
  7. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 11
  8. 2025LMM-Det: Make Large Multimodal Models Excel in Object Detectioncited by 10
  9. 2026L2P: Unlocking Latent Potential for Pixel Generationcited by 7
  10. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  11. 2025More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignmentcited by 4
  12. 2025SuperCLIP: CLIP with Simple Classification Supervisioncited by 4
38 more
  1. 2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Modelscited by 4
  2. 2026Interpretable Cross-Domain Few-Shot Learning with Rectified Target-Domain Local Alignmentcited by 4
  3. 2025A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answeringcited by 3
  4. 2025BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Modelscited by 3
  5. 2025Attention Grounded Enhancement for Visual Document Retrievalcited by 3
  6. 2025POMA-3D: The Point Map Way to 3D Scene Understandingcited by 3
  7. 2025EagleVision: A Dual-Stage Framework with BEV-grounding-based Chain-of-Thought for Spatial Intelligencecited by 3
  8. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  9. 2026SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognitioncited by 3
  10. 2025SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluationcited by 2
  11. 2025HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignmentcited by 2
  12. 2025Boosting Medical Visual Understanding From Multi-Granular Language Learningcited by 2
  13. 2025ABE-CLIP: Training-Free Attribute Binding Enhancement for Compositional Image-Text Matchingcited by 2
  14. 2026BindCLIP: A Unified Contrastive-Generative Representation Learning Framework for Virtual Screeningcited by 2
  15. 2026GeoAlignCLIP: Enhancing Fine-Grained Vision-Language Alignment in Remote Sensing via Multi-Granular Consistency Learningcited by 2
  16. 2026Uncertainty-guided Compositional Alignment with Part-to-Whole Semantic Representativeness in Hyperbolic Vision-Language Modelscited by 2
  17. 2026Mitigating Multimodal Hallucination via Phase-wise Self-rewardcited by 2
  18. 2026T-REN: Learning Text-Aligned Region Tokens Improves Dense Vision-Language Alignment and Scalabilitycited by 2
  19. 2026T-FunS3D: Task-Driven Hierarchical Open-Vocabulary 3D Functionality Segmentationcited by 2
  20. 2026TIDE: Task-Isolated Diffusion for Unified Video Editing and Generationcited by 2
  21. 2025OpenSeg-R: Improving Open-Vocabulary Segmentation via Step-by-Step Visual Reasoningcited by 1
  22. 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinationscited by 1
  23. 2025ProCLIP: Progressive Vision-Language Alignment via LLM-based Embeddercited by 1
  24. 2025FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understandingcited by 1
  25. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  26. 2025MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIPcited by 1
  27. 2025Towards Accurate UAV Image Perception: Guiding Vision-Language Models with Stronger Task Promptscited by 1
  28. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  29. 2026Universal 3D Shape Matching via Coarse-to-Fine Language Guidancecited by 1
  30. 2026Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learningcited by 1
  31. 2026Improving Reasoning in Vision-Language Models via Perception Verified Self-Trainingcited by 1
  32. 2026DialogueVPR: Towards Conversational Visual Place Recognitioncited by 1
  33. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Modelscited by 0
  34. 2025No Concept Left Behind: Test-Time Optimization for Compositional Text-to-Image Generationcited by 0
  35. 2025HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Modelscited by 0
  36. 2025RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehensioncited by 0
  37. 2026ObjEmbed: Towards Universal Multimodal Object Embeddingscited by 0
  38. 2026Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understandingcited by 0

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs
  2. 2024FLAIR: VLM with Fine-grained Language-informed Image Representations
  3. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition
  4. 2025GOAL: Global-local Object Alignment Learning
  5. 2024Improving fine-grained understanding in image-text pre-training
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training
  7. 2026FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions
  8. 2025Meta CLIP 2: A Worldwide Scaling Recipe