mlsanityData to 14 Oct 2026

On the rankability of visual embeddings

arXiv 2507.03683 ↗First posted 4 Jul 2025cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

We study whether visual embedding models capture continuous, ordinal attributes along linear directions, which we term _rank axes_. We define a model as _rankable_ for an attribute if projecting embeddings onto such an axis preserves the attribute's order. Across 7 popular encoders and 9 datasets with attributes like age, crowd count, head pose, aesthetics, and recency, we find that many embeddings are inherently rankable. Surprisingly, a small number of samples, or even just two extreme examples, often suffice to recover meaningful rank axes, without full-scale supervision. These findings open up new use cases for image ranking in vector databases and motivate further study into the structure and learning of rankable embeddings. Our code is available at https://github.com/aktsonthalia/rankable-vision-embeddings.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 55 of its 76 references that are in the map. First posted on arXiv 4 Jul 2025.

Builds on

55 of 76 references.

References · 55

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  10. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  11. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
38 more
  1. 2017Deep Image Priorcited by 1,098
  2. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  3. 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
  4. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 712
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  7. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  8. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  9. 2024The Platonic Representation Hypothesiscited by 382
  10. 2019KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessmentcited by 355
  11. 2020A Metric Learning Reality Checkcited by 321
  12. 2019Hyperbolic Image Embeddingscited by 221
  13. 2023Teaching CLIP to Count to Tencited by 198
  14. 2018Composition Loss for Counting, Density Map Estimation and Localization in Dense Crowdscited by 186
  15. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  16. 2023Hyperbolic Image-Text Representationscited by 136
  17. 2015Order-Embeddings of Images and Languagecited by 113
  18. 2022Disentangled Representation Learningcited by 112
  19. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 102
  20. 2023CLIP-Count: Towards Text-Guided Zero-Shot Object Countingcited by 83
  21. 2022Rank-N-Contrast: Learning Continuous Representations for Regressioncited by 76
  22. 2021Deep Learning for Instance Retrieval: A Surveycited by 70
  23. 2024LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIPcited by 60
  24. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  25. 2023CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Modelcited by 47
  26. 2022OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regressioncited by 44
  27. 2017Heterogeneous Face Attribute Estimation: A Deep Multi-Task Learning Approachcited by 43
  28. 2023MiVOLO: Multi-input Transformer for Age and Gender Estimationcited by 42
  29. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  30. 2024FaceXFormer: A Unified Transformer for Facial Analysiscited by 34
  31. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 31
  32. 2024It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gapcited by 26
  33. 2022Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioningcited by 25
  34. 2023Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classificationcited by 21
  35. 2024Teach CLIP to Develop a Number Sense for Ordinal Regressioncited by 20
  36. 2025Intermediate Layer Classifiers for OOD generalizationcited by 14
  37. 2024CLIP-EBC: CLIP Can Count Accurately through Enhanced Blockwise Classificationcited by 12
  38. 2023Parts of Speech-Grounded Subspaces in Vision-Language Modelscited by 11

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Does Data Scaling Lead to Visual Compositional Generalization?cited by 13
  2. 2026CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Spacecited by 1
  3. 2026Semantic Robustness Certification for Vision-Language Modelscited by 1
  4. 2026Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometrycited by 0