# Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification

arXiv: https://arxiv.org/abs/1704.04613 · HTML: https://mlsanity.com/paper/1704.04613/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2017.

## Built on by (most cited first, 9 of 9)

- [ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval](https://mlsanity.com/paper/2203.16778/) (2022)
- [Scene Text Retrieval via Joint Text Detection and Similarity Learning](https://mlsanity.com/paper/2104.01552/) (2021)
- [Knowledge Mining with Scene Text for Fine-Grained Recognition](https://mlsanity.com/paper/2203.14215/) (2022)
- [Partial Scene Text Retrieval](https://mlsanity.com/paper/2411.10261/) (2024)
- [Sequential Visual and Semantic Consistency for Semi-supervised Text Recognition](https://mlsanity.com/paper/2402.15806/) (2024)
- [Utilizing Resource-Rich Language Datasets for End-to-End Scene Text Recognition in Resource-Poor Languages](https://mlsanity.com/paper/2111.12276/) (2021)
- [MCFNet: A Multimodal Collaborative Fusion Network for Fine-Grained Semantic Classification](https://mlsanity.com/paper/2505.23365/) (2025)
- [Fine-Grained Scene Image Classification with Modality-Agnostic Adapter](https://mlsanity.com/paper/2407.02769/) (2024)
- [When Does Geometric View Synthesis Help Wine Label Retrieval? A Public One-Shot Benchmark Across Self-Supervised and Vision-Language Backbones](https://mlsanity.com/paper/2609.33359/) (2026)

## Abstract (arXiv)

Text in natural images contains rich semantics that are often highly relevant to objects or scene. In this paper, we focus on the problem of fully exploiting scene text for visual understanding. The main idea is combining word representations and deep visual features into a globally trainable deep convolutional neural network. First, the recognized words are obtained by a scene text reading system. Then, we combine the word embedding of the recognized words and the deep visual features into a single representation, which is optimized by a convolutional neural network for fine-grained image classification. In our framework, the attention mechanism is adopted to reveal the relevance between each recognized word and the given image, which further enhances the recognition performance. We have performed experiments on two datasets: Con-Text dataset and Drink Bottle dataset, that are proposed for fine-grained classification of business places and drink bottles, respectively. The experimental results consistently demonstrate that the proposed method combining textual and visual cues significantly outperforms classification with only visual representations. Moreover, we have shown that the learned representation improves the retrieval performance on the drink bottle images by a large margin, making it potentially useful in product search.