# Democratizing Fine-grained Visual Recognition with Large Language Models

arXiv: https://arxiv.org/abs/2401.13837 · HTML: https://mlsanity.com/paper/2401.13837/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 72 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 30 of 52)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Part-based R-CNNs for Fine-grained Category Detection](https://mlsanity.com/paper/1407.3867/) (2014)

## Built on by (most cited first, 25 of 25)

- [Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character](https://mlsanity.com/paper/2405.20773/) (2024)
- [Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?](https://mlsanity.com/paper/2601.06993/) (2026)
- [FungiTastic: A multi-modal dataset and benchmark for image categorization](https://mlsanity.com/paper/2408.13632/) (2024)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [Tree of Attributes Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2410.11201/) (2024)
- [Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs](https://mlsanity.com/paper/2506.03195/) (2025)
- [Vocabulary-Free 3D Instance Segmentation with Vision and Language Assistant](https://mlsanity.com/paper/2408.10652/) (2024)
- [TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration](https://mlsanity.com/paper/2410.12183/) (2024)
- [Category Discovery: An Open-World Perspective](https://mlsanity.com/paper/2509.22542/) (2025)
- [Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalability](https://mlsanity.com/paper/2503.20301/) (2025)
- [PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding](https://mlsanity.com/paper/2505.20759/) (2025)
- [Free-Grained Hierarchical Visual Recognition](https://mlsanity.com/paper/2510.14737/) (2025)
- [Open Ad-hoc Categorization with Contextualized Feature Learning](https://mlsanity.com/paper/2512.16202/) (2025)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognition](https://mlsanity.com/paper/2603.17729/) (2026)
- [Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs](https://mlsanity.com/paper/2505.01064/) (2025)
- [Multi-modal Mutual-Guidance Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2507.08410/) (2025)
- [CoAPT: Context Attribute words for Prompt Tuning](https://mlsanity.com/paper/2407.13808/) (2024)
- [Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model](https://mlsanity.com/paper/2507.23070/) (2025)
- [UniFGVC: Universal Training-Free Few-Shot Fine-Grained Vision Classification via Attribute-Aware Multimodal Retrieval](https://mlsanity.com/paper/2508.04136/) (2025)
- [Unlocking Few-Shot Capabilities in LVLMs via Prompt Conditioning and Head Selection](https://mlsanity.com/paper/2603.24181/) (2026)
- [Interpretable Text-Guided Image Clustering via Iterative Search](https://mlsanity.com/paper/2506.12514/) (2025)
- [An analysis of vision-language models for fabric retrieval](https://mlsanity.com/paper/2507.04735/) (2025)
- [Automatic Image-Level Morphological Trait Annotation for Organismal Images](https://mlsanity.com/paper/2604.01619/) (2026)
- [Bridging Coarse and Fine Recognition: A Hybrid Approach for Open-Ended Multi-Granularity Object Recognition in Interactive Educational Games](https://mlsanity.com/paper/2604.16785/) (2026)

## Abstract (arXiv)

Identifying subordinate-level categories from images is a longstanding task in computer vision and is referred to as fine-grained visual recognition (FGVR). It has tremendous significance in real-world applications since an average layperson does not excel at differentiating species of birds or mushrooms due to subtle differences among the species. A major bottleneck in developing FGVR systems is caused by the need of high-quality paired expert annotations. To circumvent the need of expert knowledge we propose Fine-grained Semantic Category Reasoning (FineR) that internally leverages the world knowledge of large language models (LLMs) as a proxy in order to reason about fine-grained category names. In detail, to bridge the modality gap between images and LLM, we extract part-level visual attributes from images as text and feed that information to a LLM. Based on the visual attributes and its internal world knowledge the LLM reasons about the subordinate-level category names. Our training-free FineR outperforms several state-of-the-art FGVR and language and vision assistant models and shows promise in working in the wild and in new domains where gathering expert annotation is arduous.