# Fine-grained Visual-textual Representation Learning

arXiv: https://arxiv.org/abs/1709.00340 · HTML: https://mlsanity.com/paper/1709.00340/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Aug 2017.

## Built on by (most cited first, 27 of 27)

- [CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification](https://mlsanity.com/paper/2204.14244/) (2022)
- [Incremental Residual Concept Bottleneck Models](https://mlsanity.com/paper/2404.08978/) (2024)
- [Introducing Visual Perception Token into Multimodal Large Language Model](https://mlsanity.com/paper/2502.17425/) (2025)
- [Semi-supervised Concept Bottleneck Models](https://mlsanity.com/paper/2406.18992/) (2024)
- [Causally Reliable Concept Bottleneck Models](https://mlsanity.com/paper/2503.04363/) (2025)
- [Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality](https://mlsanity.com/paper/2406.11048/) (2024)
- [Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning](https://mlsanity.com/paper/2104.02429/) (2021)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework](https://mlsanity.com/paper/2307.12626/) (2023)
- [Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognition](https://mlsanity.com/paper/2309.14183/) (2023)
- [DialogPaint: A Dialog-based Image Editing Model](https://mlsanity.com/paper/2303.10073/) (2023)
- [Text-Conditioned Sampling Framework for Text-to-Image Generation with Masked Generative Models](https://mlsanity.com/paper/2304.01515/) (2023)
- [Taxonomy-Structured Domain Adaptation](https://mlsanity.com/paper/2306.07874/) (2023)
- [Understanding Multimodal Deep Neural Networks: A Concept Selection View](https://mlsanity.com/paper/2404.08964/) (2024)
- [Rethinking Few Shot CLIP Benchmarks: A Critical Analysis in the Inductive Setting](https://mlsanity.com/paper/2507.20834/) (2025)
- [Unsupervised discovery of Interpretable Visual Concepts](https://mlsanity.com/paper/2309.00018/) (2023)
- [OMG-ATTACK: Self-Supervised On-Manifold Generation of Transferable Evasion Attacks](https://mlsanity.com/paper/2310.03707/) (2023)
- [MMIS: Multimodal Dataset for Interior Scene Visual Generation and Recognition](https://mlsanity.com/paper/2407.05980/) (2024)
- [CLIP Adaptation by Intra-modal Overlap Reduction](https://mlsanity.com/paper/2409.11338/) (2024)
- [ME: Trigger Element Combination Backdoor Attack on Copyright Infringement](https://mlsanity.com/paper/2506.10776/) (2025)
- [Mixture of Concept Bottleneck Experts](https://mlsanity.com/paper/2602.02886/) (2026)
- [Transforming gradient-based techniques into interpretable methods](https://mlsanity.com/paper/2401.14434/) (2024)
- [UNEM: UNrolled Generalized EM for Transductive Few-Shot Learning](https://mlsanity.com/paper/2412.16739/) (2024)
- [Enhancing Instance-Level Image Classification with Set-Level Labels](https://mlsanity.com/paper/2311.05659/) (2023)
- [DCNN: Dual Cross-current Neural Networks Realized Using An Interactive Deep Learning Discriminator for Fine-grained Objects](https://mlsanity.com/paper/2405.04093/) (2024)
- [A Concept-Centric Approach to Multi-Modality Learning](https://mlsanity.com/paper/2412.13847/) (2024)
- [ToolFG: Towards Well-Grounded Fine-Grained Image Classification](https://mlsanity.com/paper/2606.02518/) (2026)

## Abstract (arXiv)

Fine-grained visual categorization is to recognize hundreds of subcategories belonging to the same basic-level category, which is a highly challenging task due to the quite subtle and local visual distinctions among similar subcategories. Most existing methods generally learn part detectors to discover discriminative regions for better categorization performance. However, not all parts are beneficial and indispensable for visual categorization, and the setting of part detector number heavily relies on prior knowledge as well as experimental validation. As is known to all, when we describe the object of an image via textual descriptions, we mainly focus on the pivotal characteristics, and rarely pay attention to common characteristics as well as the background areas. This is an involuntary transfer from human visual attention to textual attention, which leads to the fact that textual attention tells us how many and which parts are discriminative and significant to categorization. So textual attention could help us to discover visual attention in image. Inspired by this, we propose a fine-grained visual-textual representation learning (VTRL) approach, and its main contributions are: (1) Fine-grained visual-textual pattern mining devotes to discovering discriminative visual-textual pairwise information for boosting categorization performance through jointly modeling vision and text with generative adversarial networks (GANs), which automatically and adaptively discovers discriminative parts. (2) Visual-textual representation learning jointly combines visual and textual information, which preserves the intra-modality and inter-modality information to generate complementary fine-grained representation, as well as further improves categorization performance.