# Knowledge Mining with Scene Text for Fine-Grained Recognition

arXiv: https://arxiv.org/abs/2203.14215 · HTML: https://mlsanity.com/paper/2203.14215/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 27 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Scene Text Visual Question Answering](https://mlsanity.com/paper/1905.13648/) (2019)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting](https://mlsanity.com/paper/2007.09482/) (2020)
- [Deep Joint Entity Disambiguation with Local Neural Attention](https://mlsanity.com/paper/1704.04920/) (2017)
- [Differentiable Patch Selection for Image Recognition](https://mlsanity.com/paper/2104.03059/) (2021)
- [All You Need Is Boundary: Toward Arbitrary-Shaped Text Spotting](https://mlsanity.com/paper/1911.09550/) (2019)
- [Knowledge-Aware Language Model Pretraining](https://mlsanity.com/paper/2007.00655/) (2020)
- [MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering](https://mlsanity.com/paper/2010.14095/) (2020)
- [E2E-MLT - an Unconstrained End-to-End Method for Multi-Language Scene Text](https://mlsanity.com/paper/1801.09919/) (2018)
- [Scene Text Retrieval via Joint Text Detection and Similarity Learning](https://mlsanity.com/paper/2104.01552/) (2021)
- [Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval](https://mlsanity.com/paper/2009.09809/) (2020)
- [Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification](https://mlsanity.com/paper/1704.04613/) (2017)
- [Exploiting Temporal Information for DCNN-based Fine-Grained Object Classification](https://mlsanity.com/paper/1608.00486/) (2016)
- [Fine-grained Image Classification and Retrieval by Combining Visual and Locally Pooled Textual Features](https://mlsanity.com/paper/2001.04732/) (2020)

## Built on by (most cited first, 10 of 10)

- [Red Teaming Visual Language Models](https://mlsanity.com/paper/2401.12915/) (2024)
- [Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution](https://mlsanity.com/paper/2305.07498/) (2023)
- [Ambiguous Images With Human Judgments for Robust Visual Event Classification](https://mlsanity.com/paper/2210.03102/) (2022)
- [Towards Robust Real-Time Scene Text Detection: From Semantic to Instance Representation Learning](https://mlsanity.com/paper/2308.07202/) (2023)
- [Self-distillation Regularized Connectionist Temporal Classification Loss for Text Recognition: A Simple Yet Effective Approach](https://mlsanity.com/paper/2308.08806/) (2023)
- [Partial Scene Text Retrieval](https://mlsanity.com/paper/2411.10261/) (2024)
- [Visual Crowd Analysis: Open Research Problems](https://mlsanity.com/paper/2308.10677/) (2023)
- [Cross-Level Multi-Instance Distillation for Self-Supervised Fine-Grained Visual Categorization](https://mlsanity.com/paper/2401.08860/) (2024)
- [MCFNet: A Multimodal Collaborative Fusion Network for Fine-Grained Semantic Classification](https://mlsanity.com/paper/2505.23365/) (2025)
- [Fine-Grained Scene Image Classification with Modality-Agnostic Adapter](https://mlsanity.com/paper/2407.02769/) (2024)

## Abstract (arXiv)

Recently, the semantics of scene text has been proven to be essential in fine-grained image classification. However, the existing methods mainly exploit the literal meaning of scene text for fine-grained recognition, which might be irrelevant when it is not significantly related to objects/scenes. We propose an end-to-end trainable network that mines implicit contextual knowledge behind scene text image and enhance the semantics and correlation to fine-tune the image representation. Unlike the existing methods, our model integrates three modalities: visual feature extraction, text semantics extraction, and correlating background knowledge to fine-grained image classification. Specifically, we employ KnowBert to retrieve relevant knowledge for semantic representation and combine it with image features for fine-grained classification. Experiments on two benchmark datasets, Con-Text, and Drink Bottle, show that our method outperforms the state-of-the-art by 3.72% mAP and 5.39% mAP, respectively. To further validate the effectiveness of the proposed method, we create a new dataset on crowd activity recognition for the evaluation. The source code and new dataset of this work are available at https://github.com/lanfeng4659/KnowledgeMiningWithSceneText.