mlsanityData to 7 Oct 2026

RankCLIP: Ranking-Consistent Language-Image Pretraining

arXiv 2404.09387 ↗First posted 15 Apr 2024cs.CV cs.AI cs.LGcited by 9

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Self-supervised contrastive learning models, such as CLIP, have set new benchmarks for vision-language models in many downstream tasks. However, their dependency on rigid one-to-one mappings overlooks the complex and often multifaceted relationships between and within texts and images. To this end, we introduce RankCLIP, a novel pre-training method that extends beyond the rigid one-to-one matching framework of CLIP and its variants. By extending the traditional pair-wise loss to list-wise, and leveraging both in-modal and cross-modal ranking consistency, RankCLIP improves the alignment process, enabling it to capture the nuanced many-to-many relationships between and within each modality. Through comprehensive experiments, we demonstrate the effectiveness of RankCLIP in various downstream tasks, notably achieving significant gains in zero-shot classifications over state-of-the-art methods, underscoring the importance of this enhanced learning process.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 74 references that are in the map. First posted on arXiv 15 Apr 2024.

Builds on

47 of 74 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger30 Mar
  2. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention6 Mar
  3. 2022PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining29 Apr

Other references · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  11. 2013Describing Textures in the Wildcited by 2,110
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
32 more
  1. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2019Natural Adversarial Examplescited by 1,401
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  6. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  7. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  9. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  11. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  13. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  14. 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
  15. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  16. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  17. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 171
  18. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  19. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  20. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  21. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  22. 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
  23. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  24. 2022Vision-and-Language Pretrained Models: A Surveycited by 50
  25. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  26. 2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
  27. 2023Improved Probabilistic Image-Text Representationscited by 40
  28. 2022A Review on Methods and Applications in Multimodal Deep Learningcited by 29
  29. 2022Geodesic Multi-Modal Mixup for Robust Fine-Tuningcited by 29
  30. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  31. 2021A Fistful of Words: Learning Transferable Visual Models from Bag-of-Words Supervisioncited by 19
  32. 2024Improving Medical Multi-modal Contrastive Learning with Expert Annotationscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 3; 2026: 2.

9 papers.

  1. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  2. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  3. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31
  4. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  5. 2024Beyond Training: Dynamic Token Merging for Zero-Shot Video Understandingcited by 13
  6. 2024RankByGene: Gene-Guided Histopathology Representation Learning Through Cross-Modal Ranking Consistencycited by 4
  7. 2025RLM: A Vision-Language Model Approach for Radar Scene Understandingcited by 2
  8. 2026DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistencycited by 0
  9. 2026Grad-CAM for Vision Transformers: A Systematic Taxonomy and Audit of Methodological Ambiguity in Explainable AIcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Training
  2. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression
  3. 2026Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning
  4. 2023Improved baselines for vision-language pre-training
  5. 2025HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
  6. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Models
  7. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining
  8. 2025ProCLIP: Progressive Vision-Language Alignment via LLM-based Embedder