mlsanityData to 7 Oct 2026

Jina CLIP: Your CLIP Model Is Also Your Text Retriever

arXiv 2405.20204 ↗First posted 30 May 2024cs.CL cs.AI cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) is widely used to train models to align images and texts in a common embedding space by mapping them to fixed-sized vectors. These models are key to multimodal information retrieval and related tasks. However, CLIP models generally underperform in text-only tasks compared to specialized text models. This creates inefficiencies for information retrieval systems that keep separate embeddings and models for text-only and multimodal tasks. We propose a novel, multi-task contrastive training method to address this issue, which we use to train the jina-clip-v1 model to achieve the state-of-the-art performance on both text-image and text-text retrieval tasks.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

28 of 36 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2017Decoupled Weight Decay Regularizationcited by 1,722
  10. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
16 more
  1. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  3. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  4. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  6. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  7. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  8. 2020Understanding the Behaviour of Contrastive Losscited by 397
  9. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  10. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  11. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  12. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  13. 2023Retrieving Multimodal Information for Augmented Generation: A Surveycited by 84
  14. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  15. 2024Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddingscited by 20
  16. 2023Three Towers: Flexible Contrastive Learning with Pretrained Image Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 21; 2026: 17.

46 papers.

Sort:

By area:

  1. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  2. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  3. 2025jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrievalcited by 72
  4. 2024M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Frameworkcited by 36
  5. 2025MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrievalcited by 21
  6. 2024Nomic Embed Vision: Expanding the Latent Spacecited by 19
  7. 2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 19
  8. 2024Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documentscited by 17
  9. 2025Unifying 2D and 3D Vision-Language Understandingcited by 15
  10. 2024FLAME: Frozen Large Language Models Enable Data-Efficient Language-Image Pre-trainingcited by 12
  11. 2024LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representationcited by 11
  12. 2025MIEB: Massive Image Embedding Benchmarkcited by 11
34 more
  1. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  2. 2026Nemotron ColEmbed V2: Top-Performing Late Interaction Embedding Models for Visual Document Retrievalcited by 8
  3. 2025Benchmarking Multimodal RAG through a Chart-based Document Question-Answering Generation Frameworkcited by 7
  4. 2025Distributional Vision-Language Alignment by Cauchy-Schwarz Divergencecited by 7
  5. 2025AURA: A Fine-Grained Benchmark and Decomposed Metric for Audio-Visual Reasoningcited by 7
  6. 2025AVA: Towards Agentic Video Analytics with Vision Language Modelscited by 5
  7. 2024Human Action CLIPs: Detecting AI-generated Human Motioncited by 4
  8. 2026Towards Uniformity and Alignment for Multimodal Representation Learningcited by 4
  9. 2025C-MAG: Cascade Multimodal Attributed Graphs for Supply Chain Link Predictioncited by 3
  10. 2025SafeR-CLIP: Mitigating NSFW Content in Vision-Language Models While Preserving Pre-Trained Knowledgecited by 3
  11. 2025Enhancing 3D Semantic Scene Completion with a Refinement Modulecited by 3
  12. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  13. 2026IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignmentcited by 3
  14. 2025Robustness in Both Domains: CLIP Needs a Robust Text Encodercited by 2
  15. 2025M3DR: Towards Universal Multilingual Multimodal Document Retrievalcited by 2
  16. 2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrievalcited by 2
  17. 2025Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insightscited by 1
  18. 2025Context-Adaptive Multi-Prompt Embedding with Large Language Models for Vision-Language Alignmentcited by 1
  19. 2025MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiencycited by 1
  20. 2026Reading Between the Pixels: Linking Text-Image Embedding Alignment to Typographic Attack Success on Vision-Language Modelscited by 1
  21. 2026Self Knowledge Re-expression: A Fully Local Method for Adapting LLMs to Tasks Using Intrinsic Knowledgecited by 1
  22. 2026Valley3: Scaling Omni Foundation Models for E-commercecited by 1
  23. 2025CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioningcited by 0
  24. 2025LVLM-Aware Multimodal Retrieval for RAG-Based Medical Diagnosis with General-Purpose Modelscited by 0
  25. 2025FoCLIP: A Feature-Space Misalignment Framework for CLIP-Based Image Manipulation and Detectioncited by 0
  26. 2026Multilingual-To-Multimodal (M2M): Unlocking New Languages with Monolingual Textcited by 0
  27. 2026MARVEL: Multimodal Adaptive Reasoning-intensiVe Expand-rerank and retrievaLcited by 0
  28. 2026BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignmentcited by 0
  29. 2026Adaptive Test-Time Scaling for Zero-Shot Respiratory Audio Classificationcited by 0
  30. 2026One Perturbation, Two Failure Modes: Probing VLM Safety via Embedding-Guided Typographic Perturbationscited by 0
  31. 2026Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionalitycited by 0
  32. 2026ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIPcited by 0
  33. 2026Human-Grounded Calibration for Long-Text Image-Text Congruence in Vision-Language Modelscited by 0
  34. 2026Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddingscited by 0