# Teach CLIP to Develop a Number Sense for Ordinal Regression

arXiv: https://arxiv.org/abs/2408.03574 · HTML: https://mlsanity.com/paper/2408.03574/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Teaching CLIP to Count to Ten](https://mlsanity.com/paper/2302.12066/) (2023)
- [Frozen CLIP Models are Efficient Video Learners](https://mlsanity.com/paper/2208.03550/) (2022)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [Context-Aware Crowd Counting](https://mlsanity.com/paper/1811.10452/) (2018)
- [Deep Label Distribution Learning with Label Ambiguity](https://mlsanity.com/paper/1611.01731/) (2016)
- [Can Language Understand Depth?](https://mlsanity.com/paper/2207.01077/) (2022)
- [Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression](https://mlsanity.com/paper/2103.13629/) (2021)
- [CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model](https://mlsanity.com/paper/2304.04231/) (2023)
- [Moving Window Regression: A Novel Approach to Ordinal Regression](https://mlsanity.com/paper/2203.13122/) (2022)
- [OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression](https://mlsanity.com/paper/2206.02338/) (2022)
- [BridgeNet: A Continuity-Aware Probabilistic Network for Age Estimation](https://mlsanity.com/paper/1904.03358/) (2019)
- [PML: Progressive Margin Loss for Long-tailed Age Classification](https://mlsanity.com/paper/2103.02140/) (2021)
- [No Token Left Behind: Explainability-Aided Image Classification and Generation](https://mlsanity.com/paper/2204.04908/) (2022)
- [Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification](https://mlsanity.com/paper/2306.13856/) (2023)
- [An Image is Worth More than a Thousand Favorites: Surfacing the Hidden Beauty of Flickr Pictures](https://mlsanity.com/paper/1505.03358/) (2015)
- [Ord2Seq: Regarding Ordinal Regression as Label Sequence Prediction](https://mlsanity.com/paper/2307.09004/) (2023)
- [Adaptive Contrast for Image Regression in Computer-Aided Disease Assessment](https://mlsanity.com/paper/2112.11700/) (2021)
- [Semi-Supervised Deep Regression with Uncertainty Consistency and Variational Model Ensembling via Bayesian Neural Networks](https://mlsanity.com/paper/2302.07579/) (2023)

## Built on by (most cited first, 20 of 20)

- [VisNumBench: Evaluating Number Sense of Multimodal Large Language Models](https://mlsanity.com/paper/2503.14939/) (2025)
- [CLOC: Contrastive Learning for Ordinal Classification with Multi-Margin N-pair Loss](https://mlsanity.com/paper/2504.17813/) (2025)
- [A Survey on Ordinal Regression: Applications, Advances and Prospects](https://mlsanity.com/paper/2503.00952/) (2025)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [Differential Contrastive Training for Gaze Estimation](https://mlsanity.com/paper/2502.20128/) (2025)
- [OrderChain: Towards General Instruct-Tuning for Stimulating the Ordinal Understanding Ability of MLLM](https://mlsanity.com/paper/2504.04801/) (2025)
- [Ranking-aware adapter for text-driven image ordering with CLIP](https://mlsanity.com/paper/2412.06760/) (2024)
- [TOC-UCO: a comprehensive repository of tabular ordinal classification datasets](https://mlsanity.com/paper/2507.17348/) (2025)
- [Dual-level Fuzzy Learning with Patch Guidance for Image Ordinal Regression](https://mlsanity.com/paper/2505.05834/) (2025)
- [Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models](https://mlsanity.com/paper/2512.21337/) (2025)
- [FluoCLIP: Stain-Aware Focus Quality Assessment in Fluorescence Microscopy](https://mlsanity.com/paper/2602.23791/) (2026)
- [Anny-Fit: All-Age Human Mesh Recovery](https://mlsanity.com/paper/2605.04728/) (2026)
- [STORM: Benchmarking Visual Rating of MLLMs with a Comprehensive Ordinal Regression Dataset](https://mlsanity.com/paper/2506.01738/) (2025)
- [CardiacCLIP: Video-based CLIP Adaptation for LVEF Prediction in a Few-shot Manner](https://mlsanity.com/paper/2509.17065/) (2025)
- [Contrastive Learning for Semi-Supervised Deep Regression with Generalized Ordinal Rankings from Spectral Seriation](https://mlsanity.com/paper/2512.09267/) (2025)
- [CLIP-Guided Multi-Task Regression for Multi-View Plant Phenotyping](https://mlsanity.com/paper/2603.04091/) (2026)
- [OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models](https://mlsanity.com/paper/2603.07786/) (2026)
- [VLAgeBench: Benchmarking Large Vision-Language Models for Zero-Shot Human Age Estimation](https://mlsanity.com/paper/2603.26015/) (2026)
- [Contrastive Order Learning: A General Framework for Ordinal Regression](https://mlsanity.com/paper/2607.08109/) (2026)
- [Latent Ordinal Evidence, Misaligned Outputs: Inference-Time Ordinal Lens Alignment for Multimodal LLMs](https://mlsanity.com/paper/2608.20999/) (2026)

## Abstract (arXiv)

Ordinal regression is a fundamental problem within the field of computer vision, with customised well-trained models on specific tasks. While pre-trained vision-language models (VLMs) have exhibited impressive performance on various vision tasks, their potential for ordinal regression has received less exploration. In this study, we first investigate CLIP's potential for ordinal regression, from which we expect the model could generalise to different ordinal regression tasks and scenarios. Unfortunately, vanilla CLIP fails on this task, since current VLMs have a well-documented limitation of encapsulating compositional concepts such as number sense. We propose a simple yet effective method called NumCLIP to improve the quantitative understanding of VLMs. We disassemble the exact image to number-specific text matching problem into coarse classification and fine prediction stages. We discretize and phrase each numerical bin with common language concept to better leverage the available pre-trained alignment in CLIP. To consider the inherent continuous property of ordinal regression, we propose a novel fine-grained cross-modal ranking-based regularisation loss specifically designed to keep both semantic and ordinal alignment in CLIP's feature space. Experimental results on three general ordinal regression tasks demonstrate the effectiveness of NumCLIP, with 10% and 3.83% accuracy improvement on historical image dating and image aesthetics assessment task, respectively. Code is publicly available at https://github.com/xmed-lab/NumCLIP.