# FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs

arXiv: https://arxiv.org/abs/2504.01916 · HTML: https://mlsanity.com/paper/2504.01916/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena](https://mlsanity.com/paper/2112.07566/) (2021)
- [DOCCI: Descriptions of Connected and Contrasting Images](https://mlsanity.com/paper/2404.19753/) (2024)
- [A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captions](https://mlsanity.com/paper/2312.08578/) (2023)
- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [DreamLIP: Language-Image Pre-training with Long Captions](https://mlsanity.com/paper/2403.17007/) (2024)
- [ImageInWords: Unlocking Hyper-Detailed Image Descriptions](https://mlsanity.com/paper/2405.02793/) (2024)
- [COLA: A Benchmark for Compositional Text-to-image Retrieval](https://mlsanity.com/paper/2305.03689/) (2023)
- [Improving fine-grained understanding in image-text pre-training](https://mlsanity.com/paper/2401.09865/) (2024)
- [Self-slimmed Vision Transformer](https://mlsanity.com/paper/2111.12624/) (2021)
- [LoTLIP: Improving Language-Image Pre-training for Long Text Understanding](https://mlsanity.com/paper/2410.05249/) (2024)
- [TULIP: Token-length Upgraded CLIP](https://mlsanity.com/paper/2410.10034/) (2024)

## Built on by (most cited first, 23 of 23)

- [Chat-Driven Text Generation and Interaction for Person Retrieval](https://mlsanity.com/paper/2509.12662/) (2025)
- [Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval](https://mlsanity.com/paper/2505.19650/) (2025)
- [VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions](https://mlsanity.com/paper/2508.02329/) (2025)
- [β-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment](https://mlsanity.com/paper/2512.12678/) (2025)
- [SuperCLIP: CLIP with Simple Classification Supervision](https://mlsanity.com/paper/2512.14480/) (2025)
- [Attention Grounded Enhancement for Visual Document Retrieval](https://mlsanity.com/paper/2511.13415/) (2025)
- [HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment](https://mlsanity.com/paper/2511.06653/) (2025)
- [GRAPE: Let GRPO Supervise Query Rewriting by Ranking for Retrieval](https://mlsanity.com/paper/2509.23370/) (2025)
- [FarSLIP: Discovering Effective CLIP Adaptation for Fine-Grained Remote Sensing Understanding](https://mlsanity.com/paper/2511.14901/) (2025)
- [PowerCLIP: Powerset Alignment for Contrastive Pre-Training](https://mlsanity.com/paper/2511.23170/) (2025)
- [MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP](https://mlsanity.com/paper/2512.07128/) (2025)
- [Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding](https://mlsanity.com/paper/2602.02977/) (2026)
- [TEVI: Text-Conditioned Editing of Visual Representations via Sparse Autoencoders for Improved Vision-Language Alignment](https://mlsanity.com/paper/2606.07451/) (2026)
- [CLIP Is Shortsighted: Paying Attention Beyond the First Sentence](https://mlsanity.com/paper/2602.22419/) (2026)
- [Spectrally Distilled Representations Aligned with Instruction-Augmented LLMs for Satellite Imagery](https://mlsanity.com/paper/2602.22613/) (2026)
- [Geometry-Aware CLIP Retrieval via Local Cross-Modal Alignment and Steering](https://mlsanity.com/paper/2604.16487/) (2026)
- [Fast-then-Fine: A Two-Stage Framework with Multi-Granular Representation for Cross-Modal Retrieval in Remote Sensing](https://mlsanity.com/paper/2604.20429/) (2026)
- [Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference](https://mlsanity.com/paper/2605.08814/) (2026)
- [GraSP-VL: Length as a Semantic Granularity Interface for Vision-Language Representations](https://mlsanity.com/paper/2605.17727/) (2026)
- [FAST-GOAL: Fast and Efficient Global-local Object Alignment Learning](https://mlsanity.com/paper/2605.26615/) (2026)
- [HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding](https://mlsanity.com/paper/2607.00428/) (2026)
- [Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval](https://mlsanity.com/paper/2608.18521/) (2026)
- [UNMATCH: Selective Unbalanced Token-Patch Matching for Forensic Image-Claim Verification](https://mlsanity.com/paper/2609.31766/) (2026)

## Abstract (arXiv)

As a pioneering vision-language model, CLIP (Contrastive Language-Image Pre-training) has achieved significant success across various domains and a wide range of downstream vision-language tasks. However, the text encoders in popular CLIP models are limited to processing only 77 text tokens, which constrains their ability to effectively handle longer, detail-rich captions. Additionally, CLIP models often struggle to effectively capture detailed visual and textual information, which hampers their performance on tasks that require fine-grained analysis. To address these limitations, we present a novel approach, FineLIP, that extends the capabilities of CLIP. FineLIP enhances cross-modal text-image mapping by incorporating Fine-grained alignment with Longer text input within the CLIP-style framework. FineLIP first extends the positional embeddings to handle longer text, followed by the dynamic aggregation of local image and text tokens. The aggregated results are then used to enforce fine-grained token-to-token cross-modal alignment. We validate our model on datasets with long, detailed captions across two tasks: zero-shot cross-modal retrieval and text-to-image generation. Quantitative and qualitative experimental results demonstrate the effectiveness of FineLIP, outperforming existing state-of-the-art approaches. Furthermore, comprehensive ablation studies validate the benefits of key design elements within FineLIP.