# TransHP: Image Classification with Hierarchical Prompting

arXiv: https://arxiv.org/abs/2304.06385 · HTML: https://mlsanity.com/paper/2304.06385/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [Neural Prompt Search](https://mlsanity.com/paper/2206.04673/) (2022)
- [Domain Adaptation via Prompt Learning](https://mlsanity.com/paper/2202.06687/) (2022)
- [Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework](https://mlsanity.com/paper/2204.13207/) (2022)
- [Leveraging Class Hierarchies with Metric-Guided Prototype Learning](https://mlsanity.com/paper/2007.03047/) (2020)
- [Semi-Supervised Learning with Taxonomic Labels](https://mlsanity.com/paper/2111.11595/) (2021)
- [Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification](https://mlsanity.com/paper/2302.00368/) (2023)

## Built on by (most cited first, 16 of 16)

- [Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models](https://mlsanity.com/paper/2501.15140/) (2025)
- [Democratizing Fine-grained Visual Recognition with Large Language Models](https://mlsanity.com/paper/2401.13837/) (2024)
- [ChatGPT-Powered Hierarchical Comparisons for Image Classification](https://mlsanity.com/paper/2311.00206/) (2023)
- [Hybrid Token Compression for Vision-Language Models](https://mlsanity.com/paper/2512.08240/) (2025)
- [Sample-specific Masks for Visual Reprogramming-based Prompting](https://mlsanity.com/paper/2406.03150/) (2024)
- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification](https://mlsanity.com/paper/2411.07660/) (2024)
- [Bayesian-guided Label Mapping for Visual Reprogramming](https://mlsanity.com/paper/2410.24018/) (2024)
- [SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection](https://mlsanity.com/paper/2405.10053/) (2024)
- [Attribute-based Visual Reprogramming for Vision-Language Models](https://mlsanity.com/paper/2501.13982/) (2025)
- [Tree of Attributes Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2410.11201/) (2024)
- [Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification](https://mlsanity.com/paper/2412.16780/) (2024)
- [Understanding Model Reprogramming for CLIP via Decoupling Visual Prompts](https://mlsanity.com/paper/2506.01000/) (2025)
- [Token Coordinated Prompt Attention is Needed for Visual Prompting](https://mlsanity.com/paper/2505.02406/) (2025)
- [RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs](https://mlsanity.com/paper/2603.27033/) (2026)
- [Reprogramming Vision-Language Models via Structured Prompt Reparameterization](https://mlsanity.com/paper/2609.36680/) (2026)

## Abstract (arXiv)

This paper explores a hierarchical prompting mechanism for the hierarchical image classification (HIC) task. Different from prior HIC methods, our hierarchical prompting is the first to explicitly inject ancestor-class information as a tokenized hint that benefits the descendant-class discrimination. We think it well imitates human visual recognition, i.e., humans may use the ancestor class as a prompt to draw focus on the subtle differences among descendant classes. We model this prompting mechanism into a Transformer with Hierarchical Prompting (TransHP). TransHP consists of three steps: 1) learning a set of prompt tokens to represent the coarse (ancestor) classes, 2) on-the-fly predicting the coarse class of the input image at an intermediate block, and 3) injecting the prompt token of the predicted coarse class into the intermediate feature. Though the parameters of TransHP maintain the same for all input images, the injected coarse-class prompt conditions (modifies) the subsequent feature extraction and encourages a dynamic focus on relatively subtle differences among the descendant classes. Extensive experiments show that TransHP improves image classification on accuracy (e.g., improving ViT-B/16 by +2.83% ImageNet classification accuracy), training data efficiency (e.g., +12.69% improvement under 10% ImageNet training data), and model explainability. Moreover, TransHP also performs favorably against prior HIC methods, showing that TransHP well exploits the hierarchical information. The code is available at: https://github.com/WangWenhao0716/TransHP.