# ChatGPT-Powered Hierarchical Comparisons for Image Classification

arXiv: https://arxiv.org/abs/2311.00206 · HTML: https://mlsanity.com/paper/2311.00206/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 38 of its 52 references that are in the map.
- First posted on arXiv 1 Nov 2023.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation](https://mlsanity.com/paper/2110.02711/) (2021)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Text2Mesh: Text-Driven Neural Stylization for Meshes](https://mlsanity.com/paper/2112.03221/) (2021)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://mlsanity.com/paper/2212.12017/) (2022)
- [Hierarchical Fine-Grained Image Forgery Detection and Localization](https://mlsanity.com/paper/2303.17111/) (2023)
- [Debiased Learning from Naturally Imbalanced Pseudo-Labels](https://mlsanity.com/paper/2201.01490/) (2022)

## Built on by (most cited first, 22 of 22)

- [Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models](https://mlsanity.com/paper/2403.17589/) (2024)
- [GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Recognition?](https://mlsanity.com/paper/2311.15732/) (2023)
- [An Empirical Study Into What Matters for Calibrating Vision-Language Models](https://mlsanity.com/paper/2402.07417/) (2024)
- [BiggerGait: Unlocking Gait Recognition with Layer-wise Representations from Large Vision Models](https://mlsanity.com/paper/2505.18132/) (2025)
- [ProAPO: Progressively Automatic Prompt Optimization for Visual Classification](https://mlsanity.com/paper/2502.19844/) (2025)
- [Open-Set Biometrics: Beyond Good Closed-Set Models](https://mlsanity.com/paper/2407.16133/) (2024)
- [GPT4Ego: Unleashing the Potential of Pre-trained Models for Zero-Shot Egocentric Action Recognition](https://mlsanity.com/paper/2401.10039/) (2024)
- [Enhancing Concept Localization in CLIP-based Concept Bottleneck Models](https://mlsanity.com/paper/2510.07115/) (2025)
- [BatStyler: Advancing Multi-category Style Generation for Source-free Domain Generalization](https://mlsanity.com/paper/2501.01109/) (2025)
- [AgriRegion: Region-Aware Retrieval for High-Fidelity Agricultural Advice](https://mlsanity.com/paper/2512.10114/) (2025)
- [LocalScore: Local Density-Aware Similarity Scoring for Biometrics](https://mlsanity.com/paper/2602.01012/) (2026)
- [Making Better Mistakes in CLIP-Based Zero-Shot Classification with Hierarchy-Aware Language Prompts](https://mlsanity.com/paper/2503.02248/) (2025)
- [Grounding Descriptions in Images informs Zero-Shot Visual Recognition](https://mlsanity.com/paper/2412.04429/) (2024)
- [Edge Attention Module for Object Classification](https://mlsanity.com/paper/2502.03103/) (2025)
- [Analysis of Unstructured High-Density Crowded Scenes for Crowd Monitoring](https://mlsanity.com/paper/2408.11836/) (2024)
- [Decoding Neighborhood Environments with Large Language Models](https://mlsanity.com/paper/2505.08163/) (2025)
- [Beyond Class Tokens: LLM-guided Dominant Property Mining for Few-shot Classification](https://mlsanity.com/paper/2507.20511/) (2025)
- [RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMs](https://mlsanity.com/paper/2603.27033/) (2026)
- [Multi-domain Multi-modal Document Classification Benchmark with a Multi-level Taxonomy](https://mlsanity.com/paper/2605.10550/) (2026)
- [AdaBoosting Text Prompts for Vision-Language Models](https://mlsanity.com/paper/2607.00684/) (2026)
- [Subtoken Vision Transformer for Fine-grained Recognition](https://mlsanity.com/paper/2607.09086/) (2026)
- [Position Aware Layer Queries for Test Time Training in Vision Language Models](https://mlsanity.com/paper/2609.34021/) (2026)

## Abstract (arXiv)

The zero-shot open-vocabulary challenge in image classification is tackled by pretrained vision-language models like CLIP, which benefit from incorporating class-specific knowledge from large language models (LLMs) like ChatGPT. However, biases in CLIP lead to similar descriptions for distinct but related classes, prompting our novel image classification framework via hierarchical comparisons: using LLMs to recursively group classes into hierarchies and classifying images by comparing image-text embeddings at each hierarchy level, resulting in an intuitive, effective, and explainable approach.