# OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression

arXiv: https://arxiv.org/abs/2206.02338 · HTML: https://mlsanity.com/paper/2206.02338/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 6 Jun 2022.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Fine-Grained Head Pose Estimation Without Keypoints](https://mlsanity.com/paper/1710.00925/) (2017)
- [Bounding Box Regression with Uncertainty for Accurate Object Detection](https://mlsanity.com/paper/1809.08545/) (2018)
- [Photo Aesthetics Ranking Network with Attributes and Content Adaptation](https://mlsanity.com/paper/1606.01621/) (2016)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)
- [Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression](https://mlsanity.com/paper/2103.13629/) (2021)
- [BridgeNet: A Continuity-Aware Probabilistic Network for Age Estimation](https://mlsanity.com/paper/1904.03358/) (2019)
- [Deep Regression Forests for Age Estimation](https://mlsanity.com/paper/1712.07195/) (2017)
- [Mixture Dense Regression for Object Detection and Human Pose Estimation](https://mlsanity.com/paper/1912.00821/) (2019)
- [Label Distribution Learning Forests](https://mlsanity.com/paper/1702.06086/) (2017)

## Built on by (most cited first, 30 of 44)

- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Open-Vocabulary Segmentation with Semantic-Assisted Calibration](https://mlsanity.com/paper/2312.04089/) (2023)
- [CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model](https://mlsanity.com/paper/2304.04231/) (2023)
- [4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models](https://mlsanity.com/paper/2503.10437/) (2025)
- [Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models](https://mlsanity.com/paper/2406.02915/) (2024)
- [Masked Contrastive Graph Representation Learning for Age Estimation](https://mlsanity.com/paper/2306.17798/) (2023)
- [Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification](https://mlsanity.com/paper/2306.13856/) (2023)
- [Teach CLIP to Develop a Number Sense for Ordinal Regression](https://mlsanity.com/paper/2408.03574/) (2024)
- [CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation](https://mlsanity.com/paper/2308.15226/) (2023)
- [Interpretable Vision-Language Survival Analysis with Ordinal Inductive Bias for Computational Pathology](https://mlsanity.com/paper/2409.09369/) (2024)
- [VisNumBench: Evaluating Number Sense of Multimodal Large Language Models](https://mlsanity.com/paper/2503.14939/) (2025)
- [CLIP-Gaze: Towards General Gaze Estimation via Visual-Linguistic Model](https://mlsanity.com/paper/2403.05124/) (2024)
- [CLOC: Contrastive Learning for Ordinal Classification with Multi-Margin N-pair Loss](https://mlsanity.com/paper/2504.17813/) (2025)
- [Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation](https://mlsanity.com/paper/2403.02302/) (2024)
- [LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation](https://mlsanity.com/paper/2411.08606/) (2024)
- [Tree of Attributes Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2410.11201/) (2024)
- [LANIT: Language-Driven Image-to-Image Translation for Unlabeled Data](https://mlsanity.com/paper/2208.14889/) (2022)
- [A Survey on Ordinal Regression: Applications, Advances and Prospects](https://mlsanity.com/paper/2503.00952/) (2025)
- [Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.16058/) (2025)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [OrderChain: Towards General Instruct-Tuning for Stimulating the Ordinal Understanding Ability of MLLM](https://mlsanity.com/paper/2504.04801/) (2025)
- [Boosting Medical Image-based Cancer Detection via Text-guided Supervision from Reports](https://mlsanity.com/paper/2405.14230/) (2024)
- [Ranking-aware adapter for text-driven image ordering with CLIP](https://mlsanity.com/paper/2412.06760/) (2024)
- [Parameterized Diffusion Optimization enabled Autoregressive Ordinal Regression for Diabetic Retinopathy Grading](https://mlsanity.com/paper/2507.04978/) (2025)
- [GazeCLIP: Enhancing Gaze Estimation Through Text-Guided Multimodal Learning](https://mlsanity.com/paper/2401.00260/) (2023)
- [Dual-level Fuzzy Learning with Patch Guidance for Image Ordinal Regression](https://mlsanity.com/paper/2505.05834/) (2025)
- [Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models](https://mlsanity.com/paper/2512.21337/) (2025)
- [Let's Roll a BiFTA: Bi-refinement for Fine-grained Text-visual Alignment in Vision-Language Models](https://mlsanity.com/paper/2601.20419/) (2026)
- [FluoCLIP: Stain-Aware Focus Quality Assessment in Fluorescence Microscopy](https://mlsanity.com/paper/2602.23791/) (2026)
- [DiffoR: A Unified Continuous Generative Framework for Universal Ordinal Regression](https://mlsanity.com/paper/2606.07599/) (2026)

## Abstract (arXiv)

This paper presents a language-powered paradigm for ordinal regression. Existing methods usually treat each rank as a category and employ a set of weights to learn these concepts. These methods are easy to overfit and usually attain unsatisfactory performance as the learned concepts are mainly derived from the training set. Recent large pre-trained vision-language models like CLIP have shown impressive performance on various visual tasks. In this paper, we propose to learn the rank concepts from the rich semantic CLIP latent space. Specifically, we reformulate this task as an image-language matching problem with a contrastive objective, which regards labels as text and obtains a language prototype from a text encoder for each rank. While prompt engineering for CLIP is extremely time-consuming, we propose OrdinalCLIP, a differentiable prompting method for adapting CLIP for ordinal regression. OrdinalCLIP consists of learnable context tokens and learnable rank embeddings; The learnable rank embeddings are constructed by explicitly modeling numerical continuity, resulting in well-ordered, compact language prototypes in the CLIP space. Once learned, we can only save the language prototypes and discard the huge language model, resulting in zero additional computational overhead compared with the linear head counterpart. Experimental results show that our paradigm achieves competitive performance in general ordinal regression tasks, and gains improvements in few-shot and distribution shift settings for age estimation. The code is available at https://github.com/xk-huang/OrdinalCLIP.