mlsanityData to 7 Oct 2026

Long-CLIP: Unlocking the Long-Text Capability of CLIP

arXiv 2403.15378 ↗First posted 22 Mar 2024cs.CVcited by 298

Not on the map. Why?

Cited by 89 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has been the cornerstone for zero-shot classification, text-image retrieval, and text-image generation by aligning image and text modalities. Despite its widespread adoption, a significant limitation of CLIP lies in the inadequate length of text input. The length of the text token is restricted to 77, and an empirical study shows the actual effective length is even less than 20. This prevents CLIP from handling detailed descriptions, limiting its applications for image retrieval and text-to-image generation with extensive prerequisites. To this end, we propose Long-CLIP as a plug-and-play alternative to CLIP that supports long-text input, retains or even surpasses its zero-shot generalizability, and aligns the CLIP latent space, making it readily replace CLIP without any further adaptation in downstream frameworks. Nevertheless, achieving this goal is far from straightforward, as simplistic fine-tuning can result in a significant degradation of CLIP's performance. Moreover, substituting the text encoder with a language model supporting longer contexts necessitates pretraining with vast amounts of data, incurring significant expenses. Accordingly, Long-CLIP introduces an efficient fine-tuning solution on CLIP with two novel strategies designed to maintain the original capabilities, including (1) a knowledge-preserved stretching of positional embedding and (2) a primary component matching of CLIP features. With leveraging just one million extra long text-image pairs, Long-CLIP has shown the superiority to CLIP for about 20% in long caption text-image retrieval and 6% in traditional text-image retrieval tasks, e.g., COCO and Flickr30k. Furthermore, Long-CLIP offers enhanced capabilities for generating images from detailed text descriptions by replacing CLIP in a plug-and-play manner.

Key facts. Cited by 298 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 22 Mar 2024.

Builds on

28 of 36 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  11. 2019Natural Adversarial Examplescited by 1,401
  12. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
16 more
  1. 2021Grounded Language-Image Pre-trainingcited by 1,147
  2. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2022Language-driven Semantic Segmentationcited by 711
  6. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  7. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  8. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  9. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  10. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  11. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  12. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  13. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  14. 2023Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformerscited by 91
  15. 2022Position-guided Text Prompt for Vision-Language Pre-trainingcited by 27
  16. 2022When are Lemons Purple? The Concept Association Bias of Vision-Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 61; 2025: 151; 2026: 86.

298 papers.

Sort:

By area:

  1. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  2. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  3. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 206
  4. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 203
  5. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  6. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  7. 2025HiDream-I1: A High-Efficient Image Generative Foundation Model with Sparse Diffusion Transformercited by 131
  8. 2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 123
  9. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  10. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  11. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  12. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
38 more
  1. 2024What If We Recaption Billions of Web Images with LLaMA-3?cited by 82
  2. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 78
  3. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  4. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  5. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  6. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  7. 2024Faster Diffusion via Temporal Attention Decompositioncited by 54
  8. 2024Unleashing Generalization of End-to-End Autonomous Driving with Controllable Long Video Generationcited by 54
  9. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  10. 2024FLAIR: VLM with Fine-grained Language-informed Image Representationscited by 47
  11. 2024Jina CLIP: Your CLIP Model Is Also Your Text Retrievercited by 46
  12. 2024VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modelingcited by 42
  13. 2025UniME-V2: MLLM-as-a-Judge for Universal Multimodal Embedding Learningcited by 41
  14. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 40
  15. 2024Where am I? Cross-View Geo-localization with Natural Language Descriptionscited by 38
  16. 2024Diffusion Feedback Helps CLIP See Bettercited by 37
  17. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 36
  18. 2025CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learningcited by 36
  19. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  20. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  21. 2025Chat-Driven Text Generation and Interaction for Person Retrievalcited by 34
  22. 2024AIGIQA-20K: A Large Database for AI-Generated Image Quality Assessmentcited by 33
  23. 2025VideoChat-A1: Thinking with Long Videos by Chain-of-Shot Reasoningcited by 32
  24. 2024Reason-before-Retrieve: One-Stage Reflective Chain-of-Thoughts for Training-Free Zero-Shot Composed Image Retrievalcited by 31
  25. 2024FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERscited by 30
  26. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 30
  27. 2024LoTLIP: Improving Language-Image Pre-training for Long Text Understandingcited by 27
  28. 2025Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Miningcited by 27
  29. 2025SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesiscited by 26
  30. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  31. 2024LHRS-Bot-Nova: Improved Multimodal Large Language Model for Remote Sensing Vision-Language Interpretationcited by 25
  32. 2024Personalized Visual Instruction Tuningcited by 24
  33. 2025Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparitiescited by 24
  34. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  35. 2024MLLM-as-a-Judge for Image Safety without Human Labelingcited by 23
  36. 2025FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputscited by 23
  37. 2025HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generationcited by 22
  38. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 20

50 most cited of 298 All 298 (JSON)