# Fine-graind Image Classification via Combining Vision and Language

arXiv: https://arxiv.org/abs/1704.02792 · HTML: https://mlsanity.com/paper/1704.02792/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 30 of 51)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models](https://mlsanity.com/paper/2209.15639/) (2022)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Waffling around for Performance: Visual Classification with Random Words and Broad Concepts](https://mlsanity.com/paper/2306.07282/) (2023)
- [Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2305.07011/) (2023)
- [Compositional Entailment Learning for Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2410.06912/) (2024)
- [Open-Set Image Tagging with Multi-Grained Text Supervision](https://mlsanity.com/paper/2310.15200/) (2023)
- [VL-LTR: Learning Class-wise Visual-Linguistic Representation for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2111.13579/) (2021)
- [Contrastive Feature Masking Open-Vocabulary Vision Transformer](https://mlsanity.com/paper/2309.00775/) (2023)
- [Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models](https://mlsanity.com/paper/2501.15140/) (2025)
- [MetaFormer: A Unified Meta Framework for Fine-Grained Recognition](https://mlsanity.com/paper/2203.02751/) (2022)
- [Democratizing Fine-grained Visual Recognition with Large Language Models](https://mlsanity.com/paper/2401.13837/) (2024)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Robust Visual Question Answering: Datasets, Methods, and Future Challenges](https://mlsanity.com/paper/2307.11471/) (2023)
- [SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization](https://mlsanity.com/paper/2209.02109/) (2022)
- [Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images](https://mlsanity.com/paper/2311.01064/) (2023)
- [Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning](https://mlsanity.com/paper/2104.02429/) (2021)
- [Hephaestus: A large scale multitask dataset towards InSAR understanding](https://mlsanity.com/paper/2204.09435/) (2022)
- [Predictive Chemistry Augmented with Text Retrieval](https://mlsanity.com/paper/2312.04881/) (2023)
- [Beyond DAGs: A Latent Partial Causal Model for Multimodal Learning](https://mlsanity.com/paper/2402.06223/) (2024)
- [The Curious Layperson: Fine-Grained Image Recognition without Expert Labels](https://mlsanity.com/paper/2111.03651/) (2021)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [CILP-FGDI: Exploiting Vision-Language Model for Generalizable Person Re-Identification](https://mlsanity.com/paper/2501.16065/) (2025)
- [PHyCLIP: ℓ1-Product of Hyperbolic Factors Unifies Hierarchy and Compositionality in Vision-Language Representation Learning](https://mlsanity.com/paper/2510.08919/) (2025)
- [Making a Bird AI Expert Work for You and Me](https://mlsanity.com/paper/2112.02747/) (2021)
- [Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions](https://mlsanity.com/paper/2207.03133/) (2022)
- [Multi-Granularity Archaeological Dating of Chinese Bronze Dings Based on a Knowledge-Guided Relation Graph](https://mlsanity.com/paper/2303.15266/) (2023)

## Abstract (arXiv)

Fine-grained image classification is a challenging task due to the large intra-class variance and small inter-class variance, aiming at recognizing hundreds of sub-categories belonging to the same basic-level category. Most existing fine-grained image classification methods generally learn part detection models to obtain the semantic parts for better classification accuracy. Despite achieving promising results, these methods mainly have two limitations: (1) not all the parts which obtained through the part detection models are beneficial and indispensable for classification, and (2) fine-grained image classification requires more detailed visual descriptions which could not be provided by the part locations or attribute annotations. For addressing the above two limitations, this paper proposes the two-stream model combining vision and language (CVL) for learning latent semantic representations. The vision stream learns deep representations from the original visual information via deep convolutional neural network. The language stream utilizes the natural language descriptions which could point out the discriminative parts or characteristics for each image, and provides a flexible and compact way of encoding the salient visual aspects for distinguishing sub-categories. Since the two streams are complementary, combining the two streams can further achieves better classification accuracy. Comparing with 12 state-of-the-art methods on the widely used CUB-200-2011 dataset for fine-grained image classification, the experimental results demonstrate our CVL approach achieves the best performance.