# More Grounded Image Captioning by Distilling Image-Text Matching Model

arXiv: https://arxiv.org/abs/2004.00390 · HTML: https://mlsanity.com/paper/2004.00390/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Apr 2020.

## Built on by (most cited first, 30 of 43)

- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling](https://mlsanity.com/paper/2111.12085/) (2021)
- [METransformer: Radiology Report Generation by Transformer with Multiple Learnable Expert Tokens](https://mlsanity.com/paper/2304.02211/) (2023)
- [Less is More: Generating Grounded Navigation Instructions from Landmarks](https://mlsanity.com/paper/2111.12872/) (2021)
- [Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editing](https://mlsanity.com/paper/2412.19806/) (2024)
- [Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation](https://mlsanity.com/paper/2206.01988/) (2022)
- [ChatterBox: Multi-round Multimodal Referring and Grounding](https://mlsanity.com/paper/2401.13307/) (2024)
- [Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning](https://mlsanity.com/paper/2110.11767/) (2021)
- [CAMANet: Class Activation Map Guided Attention Network for Radiology Report Generation](https://mlsanity.com/paper/2211.01412/) (2022)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)
- [IDEAL: Query-Efficient Data-Free Learning from Black-box Models](https://mlsanity.com/paper/2205.11158/) (2022)
- [Semi-Autoregressive Transformer for Image Captioning](https://mlsanity.com/paper/2106.09436/) (2021)
- [Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation](https://mlsanity.com/paper/2307.13368/) (2023)
- [Localized Symbolic Knowledge Distillation for Visual Commonsense Models](https://mlsanity.com/paper/2312.04837/) (2023)
- [Grounding-IQA: Grounding Multimodal Language Model for Image Quality Assessment](https://mlsanity.com/paper/2411.17237/) (2024)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Revisiting Local Descriptor for Improved Few-Shot Classification](https://mlsanity.com/paper/2103.16009/) (2021)
- [Image Captioning via Compact Bidirectional Architecture](https://mlsanity.com/paper/2201.01984/) (2022)
- [Image Search with Text Feedback by Additive Attention Compositional Learning](https://mlsanity.com/paper/2203.03809/) (2022)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [Do DALL-E and Flamingo Understand Each Other?](https://mlsanity.com/paper/2212.12249/) (2022)
- [Exploring Multi-Modal Contextual Knowledge for Open-Vocabulary Object Detection](https://mlsanity.com/paper/2308.15846/) (2023)
- [Lesion Guided Explainable Few Weak-shot Medical Report Generation](https://mlsanity.com/paper/2211.08732/) (2022)
- [ALADIN: Distilling Fine-grained Alignment Scores for Efficient Image-Text Matching and Retrieval](https://mlsanity.com/paper/2207.14757/) (2022)
- [What You Say Is What You Show: Visual Narration Detection in Instructional Videos](https://mlsanity.com/paper/2301.02307/) (2023)
- [iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability](https://mlsanity.com/paper/2107.10300/) (2021)
- [Cross-modal Contrastive Distillation for Instructional Activity Anticipation](https://mlsanity.com/paper/2201.06734/) (2022)
- [Top-Down Framework for Weakly-supervised Grounded Image Captioning](https://mlsanity.com/paper/2306.07490/) (2023)

## Abstract (arXiv)

Visual attention not only improves the performance of image captioners, but also serves as a visual interpretation to qualitatively measure the caption rationality and model transparency. Specifically, we expect that a captioner can fix its attentive gaze on the correct objects while generating the corresponding words. This ability is also known as grounded image captioning. However, the grounding accuracy of existing captioners is far from satisfactory. To improve the grounding accuracy while retaining the captioning quality, it is expensive to collect the word-region alignment as strong supervision. To this end, we propose a Part-of-Speech (POS) enhanced image-text matching model (SCAN citelee2018stacked): POS-SCAN, as the effective knowledge distillation for more grounded image captioning. The benefits are two-fold: 1) given a sentence and an image, POS-SCAN can ground the objects more accurately than SCAN; 2) POS-SCAN serves as a word-region alignment regularization for the captioner's visual attention module. By showing benchmark experimental results, we demonstrate that conventional image captioners equipped with POS-SCAN can significantly improve the grounding accuracy without strong supervision. Last but not the least, we explore the indispensable Self-Critical Sequence Training (SCST) citeRennie2017CVPR in the context of grounded image captioning and show that the image-text matching score can serve as a reward for more grounded captioning footnotehttps://github.com/YuanEZhou/Grounded-Image-Captioning.