mlsanityData to 14 Oct 2026

FLAIR: VLM with Fine-grained Language-informed Image Representations

arXiv 2412.03561 ↗First posted 4 Dec 2024cs.CV cs.AIcited by 48

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

CLIP has shown impressive results in aligning images and texts at scale. However, its ability to capture detailed visual features remains limited because CLIP matches images and texts at a global level. To address this issue, we propose FLAIR, Fine-grained Language-informed Image Representations, an approach that utilizes long and detailed image descriptions to learn localized image embeddings. By sampling diverse sub-captions that describe fine-grained details about an image, we train our vision-language model to produce not only global embeddings but also text-specific image representations. Our model introduces text-conditioned attention pooling on top of local image tokens to produce fine-grained image representations that excel at retrieving detailed image content. We achieve state-of-the-art performance on both, existing multimodal retrieval benchmarks, as well as, our newly introduced fine-grained retrieval task which evaluates vision-language models' ability to retrieve partial image content. Furthermore, our experiments demonstrate the effectiveness of FLAIR trained on 30M image-text pairs in capturing fine-grained visual information, including zero-shot semantic segmentation, outperforming models trained on billions of pairs. Code is available at https://github.com/ExplainableML/flair .

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 63 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

47 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024TULIP: Token-length Upgraded CLIP13 Oct
  2. 2024ClearCLIP: Decomposing CLIP Representations for Dense Vision-Language Inference17 Jul
  3. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining30 Apr
  4. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger30 Mar
  5. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-training27 Sep
  6. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining29 Apr

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2013Describing Textures in the Wildcited by 2,125
  11. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  12. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
29 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  3. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  6. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  7. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  8. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 864
  9. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  10. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  12. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  13. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  14. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  15. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 300
  16. 2023Improving CLIP Training with Language Rewritescited by 217
  17. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  18. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  19. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 170
  20. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 132
  21. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  22. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  23. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  24. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  25. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 57
  26. 2024From Pixels to Prose: A Large Dataset of Dense Image Captionscited by 54
  27. 2022Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervisioncited by 36
  28. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  29. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 28.

48 papers.

Sort:

By area:

  1. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  2. 2025FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Textcited by 7
  3. 2025Explaining Similarity in Vision-Language Encoders with Weighted Banzhaf Interactionscited by 6
  4. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  5. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  6. 2025β-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignmentcited by 5
  7. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  8. 2025PathFLIP: Fine-grained Language-Image Pretraining for Versatile Computational Pathologycited by 4
  9. 2026FINER: MLLMs Hallucinate under Fine-grained Negative Queriescited by 4
  10. 2026Steerable Visual Representationscited by 4
  11. 2025Sparse Autoencoders are Topic Modelscited by 3
  12. 2025A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedbackcited by 3
36 more
  1. 2026Adaptive Global and Fine-Grained Perceptual Fusion for MLLM Embeddings Compatible with Hard Negative Amplificationcited by 3
  2. 2025Learning complete and explainable visual representations from itemized text supervisioncited by 2
  3. 2026Rethinking Concept Bottleneck Models: From Pitfalls to Solutionscited by 2
  4. 2025DALIP: Distribution Alignment-based Language-Image Pre-Training for Domain-Specific Datacited by 1
  5. 2025FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attentioncited by 1
  6. 2025FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understandingcited by 1
  7. 2025MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIPcited by 1
  8. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  9. 2026Better with Less: Tackling Heterogeneous Multi-Modal Image Joint Pretraining via Conditioned and Degraded Masked Autoencodercited by 1
  10. 2026Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learningcited by 1
  11. 2026Vision Harnessing Agent for Open Ad-hoc Segmentationcited by 1
  12. 2026TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignmentcited by 1
  13. 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perceptioncited by 1
  14. 2026DialogueVPR: Towards Conversational Visual Place Recognitioncited by 1
  15. 2025NeuroLIP: Interpretable and Fair Cross-Modal Alignment of fMRI and Phenotypic Textcited by 0
  16. 2025Mining Contextualized Visual Associations from Images for Creativity Understandingcited by 0
  17. 2025Multi-Sequence Parotid Gland Lesion Segmentation via Expert Text-Guided Segment Anything Modelcited by 0
  18. 2025BcQLM: Efficient Vision-Language Understanding with Distilled Q-Gated Cross-Modal Fusioncited by 0
  19. 2025microCLIP: Unsupervised CLIP Adaptation via Coarse-Fine Token Fusion for Fine-Grained Image Classificationcited by 0
  20. 2026Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understandingcited by 0
  21. 2026CLIP Is Shortsighted: Paying Attention Beyond the First Sentencecited by 0
  22. 2026ITO: Multi-View Alignment and Training-Time Fusion for Image-Text Pretrainingcited by 0
  23. 2026LoFi: Location-Aware Fine-Grained Representation Learning for Chest X-raycited by 0
  24. 2026GoldiCLIP: The Goldilocks Approach for Balancing Explicit Supervision for Language-Image Pretrainingcited by 0
  25. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0
  26. 2026Rule-VLN: Bridging Perception and Compliance via Semantic Reasoning and Geometric Rectificationcited by 0
  27. 2026AFMRL: Attribute-Enhanced Fine-Grained Multi-Modal Representation Learning in E-commercecited by 0
  28. 2026LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignmentcited by 0
  29. 2026Subtoken Vision Transformer for Fine-grained Recognitioncited by 0
  30. 2026Attention from Above: A Multimodal Model for Drone-Based Object Localizationcited by 0
  31. 2026Achieving Text-based Person Retrieval with Any Granularitycited by 0
  32. 2026Unlocking Spatial Grounding in Large Audio-Visual Retrieval modelscited by 0
  33. 2026RECOUNT: Reference-guided Counting with Synthetic Visual Exemplarscited by 0
  34. 2026ProtoLIP: From Sentence-Level to Object-Level Evidence Disentanglementcited by 0
  35. 2026RegRet: Enhancing Region-Level Retrieval in Large Multimodal Modelscited by 0
  36. 2026Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FG-CLIP: Fine-Grained Visual and Textual Alignment
  2. 2025Language-driven Fine-grained Retrieval
  3. 2024Improving fine-grained understanding in image-text pre-training
  4. 2026A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
  5. 2025ABC: Achieving Better Control of Multimodal Embeddings using VLMs
  6. 2024Finetuning CLIP to Reason about Pairwise Differences
  7. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition
  8. 2026State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models