# Text and Image Are Mutually Beneficial: Enhancing Training-Free Few-Shot Classification with CLIP

arXiv: https://arxiv.org/abs/2412.11375 · HTML: https://mlsanity.com/paper/2412.11375/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Do ImageNet Classifiers Generalize to ImageNet?](https://mlsanity.com/paper/1902.10811/) (2019)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation](https://mlsanity.com/paper/2110.02624/) (2021)
- [Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners](https://mlsanity.com/paper/2303.02151/) (2023)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP](https://mlsanity.com/paper/2404.02285/) (2024)
- [A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation](https://mlsanity.com/paper/2402.04087/) (2024)
- [VT-CLIP: Enhancing Vision-Language Models with Visual-guided Texts](https://mlsanity.com/paper/2112.02399/) (2021)
- [ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization](https://mlsanity.com/paper/2010.03276/) (2020)

## Built on by (most cited first, 6 of 6)

- [Backpropagation-Free Test-Time Adaptation via Probabilistic Gaussian Alignment](https://mlsanity.com/paper/2508.15568/) (2025)
- [When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging](https://mlsanity.com/paper/2602.05536/) (2026)
- [Towards Robust Text-to-Image Person Retrieval: Multi-View Reformulation for Semantic Compensation](https://mlsanity.com/paper/2604.18376/) (2026)
- [Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding](https://mlsanity.com/paper/2603.07625/) (2026)
- [Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection](https://mlsanity.com/paper/2603.24181/) (2026)
- [NutriScreener: Retrieval-Augmented Multi-Pose Graph Attention Network for Malnourishment Screening](https://mlsanity.com/paper/2511.16566/) (2025)

## Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) has been widely used in vision tasks. Notably, CLIP has demonstrated promising performance in few-shot learning (FSL). However, existing CLIP-based methods in training-free FSL (i.e., without the requirement of additional training) mainly learn different modalities independently, leading to two essential issues: 1) severe anomalous match in image modality; 2) varying quality of generated text prompts. To address these issues, we build a mutual guidance mechanism, that introduces an Image-Guided-Text (IGT) component to rectify varying quality of text prompts through image representations, and a Text-Guided-Image (TGI) component to mitigate the anomalous match of image modality through text representations. By integrating IGT and TGI, we adopt a perspective of Text-Image Mutual guidance Optimization, proposing TIMO. Extensive experiments show that TIMO significantly outperforms the state-of-the-art (SOTA) training-free method. Additionally, by exploring the extent of mutual guidance, we propose an enhanced variant, TIMO-S, which even surpasses the best training-required methods by 0.33% with approximately 100 times less time cost. Our code is available at https://github.com/lyymuwu/TIMO.