# Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation

arXiv: https://arxiv.org/abs/2311.01034 · HTML: https://mlsanity.com/paper/2311.01034/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 41)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [Prompt Distribution Learning](https://mlsanity.com/paper/2205.03340/) (2022)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos](https://mlsanity.com/paper/1811.06152/) (2018)
- [Learning Visual Representations with Caption Annotations](https://mlsanity.com/paper/2008.01392/) (2020)
- [DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation](https://mlsanity.com/paper/2203.14211/) (2022)
- [Style Augmentation: Data Augmentation via Style Randomization](https://mlsanity.com/paper/1809.05375/) (2018)
- [Can Language Understand Depth?](https://mlsanity.com/paper/2207.01077/) (2022)
- [Predicting Deep Zero-Shot Convolutional Neural Networks using Textual Descriptions](https://mlsanity.com/paper/1506.00511/) (2015)
- [Learning monocular depth estimation infusing traditional stereo knowledge](https://mlsanity.com/paper/1904.04144/) (2019)
- [PLOT: Prompt Learning with Optimal Transport for Vision-Language Models](https://mlsanity.com/paper/2210.01253/) (2022)
- [Multi-modal Alignment using Representation Codebook](https://mlsanity.com/paper/2203.00048/) (2022)
- [Self-supervised learning of visual features through embedding images into text topic spaces](https://mlsanity.com/paper/1705.08631/) (2017)

## Built on by (most cited first, 20 of 20)

- [Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models](https://mlsanity.com/paper/2410.12790/) (2024)
- [WorDepth: Variational Language Prior for Monocular Depth Estimation](https://mlsanity.com/paper/2404.03635/) (2024)
- [RSA: Resolving Scale Ambiguities in Monocular Depth Estimators through Language Descriptions](https://mlsanity.com/paper/2410.02924/) (2024)
- [Vision-Language Embodiment for Monocular Depth Estimation](https://mlsanity.com/paper/2503.16535/) (2025)
- [ScaleDepth: Decomposing Metric Depth Estimation into Scale Prediction and Relative Depth Estimation](https://mlsanity.com/paper/2407.08187/) (2024)
- [Enhancing Vision-Language Few-Shot Adaptation with Negative Learning](https://mlsanity.com/paper/2403.12964/) (2024)
- [CLIP Can Understand Depth](https://mlsanity.com/paper/2402.03251/) (2024)
- [Conceptual Codebook Learning for Vision-Language Models](https://mlsanity.com/paper/2407.02350/) (2024)
- [Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2510.09320/) (2025)
- [Ranking-aware adapter for text-driven image ordering with CLIP](https://mlsanity.com/paper/2412.06760/) (2024)
- [CLIPSym: Delving into Symmetry Detection with CLIP](https://mlsanity.com/paper/2508.14197/) (2025)
- [SPACE-CLIP: Spatial Perception via Adaptive CLIP Embeddings for Monocular Depth Estimation](https://mlsanity.com/paper/2601.17657/) (2026)
- [TR2M: Transferring Monocular Relative Depth to Metric Depth with Language Descriptions and Dual-Level Scale-Oriented Contrast](https://mlsanity.com/paper/2506.13387/) (2025)
- [PureCLIP-Depth: Prompt-Free and Decoder-Free Monocular Depth Estimation within CLIP Embedding Space](https://mlsanity.com/paper/2603.16238/) (2026)
- [Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset](https://mlsanity.com/paper/2404.05183/) (2024)
- [Towards Optimal Aggregation of Varying Range Dependencies in Haze Removal](https://mlsanity.com/paper/2408.12317/) (2024)
- [VGLD: Visually-Guided Linguistic Disambiguation for Monocular Depth Scale Recovery](https://mlsanity.com/paper/2505.02704/) (2025)
- [RAD: Retrieval-Augmented Monocular Metric Depth Estimation for Underrepresented Classes](https://mlsanity.com/paper/2602.09532/) (2026)
- [Lightweight Prompt-Guided CLIP Adaptation for Monocular Depth Estimation](https://mlsanity.com/paper/2604.01118/) (2026)
- [SPACE-CLIPv2: Decoding Local Geometry from Frozen CLIP for Monocular Depth Estimation](https://mlsanity.com/paper/2610.05029/) (2026)

## Abstract (arXiv)

Pre-trained Vision-Language Models (VLMs), such as CLIP, have shown enhanced performance across a range of tasks that involve the integration of visual and linguistic modalities. When CLIP is used for depth estimation tasks, the patches, divided from the input images, can be combined with a series of semantic descriptions of the depth information to obtain similarity results. The coarse estimation of depth is then achieved by weighting and summing the depth values, called depth bins, corresponding to the predefined semantic descriptions. The zero-shot approach circumvents the computational and time-intensive nature of traditional fully-supervised depth estimation methods. However, this method, utilizing fixed depth bins, may not effectively generalize as images from different scenes may exhibit distinct depth distributions. To address this challenge, we propose a few-shot-based method which learns to adapt the VLMs for monocular depth estimation to balance training costs and generalization capabilities. Specifically, it assigns different depth bins for different scenes, which can be selected by the model during inference. Additionally, we incorporate learnable prompts to preprocess the input text to convert the easily human-understood text into easily model-understood vectors and further enhance the performance. With only one image per scene for training, our extensive experiment results on the NYU V2 and KITTI dataset demonstrate that our method outperforms the previous state-of-the-art method by up to 10.6% in terms of MARE.