mlsanityData to 7 Oct 2026

TULIP: Token-length Upgraded CLIP

arXiv 2410.10034 ↗First posted 13 Oct 2024cs.CVcited by 18

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

We address the challenge of representing long captions in vision-language models, such as CLIP. By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens and hindering performance on tasks requiring longer descriptions. Although recent work has attempted to overcome this limit, their proposed approaches struggle to model token relationships over longer distances and simply extend to a fixed new token length. Instead, we propose a generalizable method, named TULIP, able to upgrade the token length to any length for CLIP-like models. We do so by improving the architecture with relative position encodings, followed by a training procedure that (i) distills the original CLIP text encoder into an encoder with relative position encodings and (ii) enhances the model for aligning longer captions with images. By effectively encoding captions longer than the default 77 tokens, our model outperforms baselines on cross-modal tasks such as retrieval and text-to-image generation. The code repository is available at https://github.com/ivonajdenkoska/tulip.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 42 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

39 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2024The Llama 3 Herd of Modelscited by 14,192
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
26 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  4. 2024Mixtral of Expertscited by 1,785
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  11. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  13. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  14. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  15. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  16. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  17. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  18. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  19. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  20. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  21. 2023MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Trainingcited by 96
  22. 2024DreamLIP: Language-Image Pre-training with Long Captionscited by 72
  23. 2024Contextual Position Encoding: Learning to Count What's Importantcited by 53
  24. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  25. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  26. 2024CLIP with Quality Captions: A Strong Pretraining for Vision Taskscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 5.

18 papers.

Sort:

By area:

  1. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  2. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  3. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20
  4. 2025SmartCLIP: Modular Vision-language Alignment with Identification Guaranteescited by 14
  5. 2025Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentationcited by 10
  6. 2025FIX-CLIP: Dual-Branch Hierarchical Contrastive Learning via Synthetic Captions for Better Understanding of Long Textcited by 7
  7. 2025Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillationcited by 6
  8. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  9. 2025β-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignmentcited by 5
  10. 2025DGTRSD & DGTRS-CLIP: A Dual-Granularity Remote Sensing Image-Text Dataset and Vision Language Foundation Model for Alignmentcited by 4
  11. 2025Dense Motion Captioningcited by 2
  12. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptionscited by 1
6 more
  1. 2025MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIPcited by 1
  2. 2026Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understandingcited by 1
  3. 2026TuringViT: Making SOTA Vision Transformers Accessible to Allcited by 1
  4. 2026CLIP Is Shortsighted: Paying Attention Beyond the First Sentencecited by 0
  5. 2026HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understandingcited by 0
  6. 2026Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TULIP: Towards Unified Language-Image Pretraining
  2. 2026A Paragraph is Worth a Thousand Captions: Rethinking Text Supervision for Vision-Language Retrieval
  3. 2023Text encoders bottleneck compositionality in contrastive vision-language models
  4. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encoder
  5. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  6. 2023VeCLIP: Improving CLIP Training via Visual-enriched Captions
  7. 2023Linear Alignment of Vision-language Models for Image Captioning
  8. 2025LongProLIP: A Probabilistic Vision-Language Model with Long Context Text