Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP
Abstract (arXiv)
Recent research has shown that aligning fine-grained text descriptions with localized image patches can significantly improve the zero-shot performance of pre-trained vision-language models (e.g., CLIP). However, we find that both fine-grained text descriptions and localized image patches often contain redundant information, making text-visual alignment less effective. In this paper, we tackle this issue from two perspectives: View Refinement and Description refinement, termed as textitBi-refinement for Fine-grained Text-visual Alignment (BiFTA). View refinement removes redundant image patches with high Intersection over Union (IoU) ratios, resulting in more distinctive visual samples. Description refinement removes redundant text descriptions with high pairwise cosine similarity, ensuring greater diversity in the remaining descriptions. BiFTA achieves superior zero-shot performance on 6 benchmark datasets for both ViT-based and ResNet-based CLIP, justifying the necessity to remove redundant information in visual-text alignment.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 52 references that are in the map. First posted on arXiv 28 Jan 2026.
Builds on
42 of 52 references.
Closest ancestors · 10
Same direction, newest first
- 2025Constrained Prompt Enhancement for Improving Zero-Shot Generalization of Vision-Language Models24 Aug
- 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model30 Jun
- 2025From Local Details to Global Context: Advancing Vision-Language Models with Attention-Based Selection19 May
- 2025Generalizable Prompt Learning of CLIP: A Brief Overview3 Mar
- 2024Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models5 Jun
- 2023Waffling around for Performance: Visual Classification with Random Words and Broad Concepts12 Jun
- 2022Visual Classification via Description from Large Language Models13 Oct
- 2022Learning to Decompose Visual Features with Latent Textual Prompts9 Oct
- 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models15 Sep
- 2021Learning to Prompt for Vision-Language Models2 Sep
Other references · 32
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2020Language Models are Few-Shot Learnerscited by 32,320
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,363
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
- 2013Describing Textures in the Wildcited by 2,115
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,267
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,227
- 2021Florence: A New Foundation Model for Computer Visioncited by 803
20 more
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 648
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
- 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 531
- 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
- 2024LightRAG: Simple and Fast Retrieval-Augmented Generationcited by 311
- 2022What does a platypus look like? Generating customized prompts for zero-shot image classificationcited by 264
- 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 201
- 2024RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generationcited by 144
- 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 84
- 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
- 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
- 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
- 2023π-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolationcited by 40
- 2020ALICE: Active Learning with Contrastive Natural Language Explanationscited by 36
- 2023POUF: Prompt-oriented unsupervised fine-tuning for large pre-trained modelscited by 29
- 2024Sample-specific Masks for Visual Reprogramming-based Promptingcited by 16
- 2024Bayesian-guided Label Mapping for Visual Reprogrammingcited by 12
- 2025Understanding Model Reprogramming for CLIP via Decoupling Visual Promptscited by 4
- 2025Neural Network Reprogrammability: A Unified Theme on Model Reprogramming, Prompt Tuning, and Prompt Instructioncited by 3
- 2025Attention Guided Alignment in Efficient Vision-Language Modelscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2023Mismatch Quest: Visual and Textual Feedback for Image-Text Misalignment
- 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs
- 2024ComAlign: Compositional Alignment in Vision-Language Models
- 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment
- 2026TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
- 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding
- 2025MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP
- 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification