# Show, Edit and Tell: A Framework for Editing Image Captions

arXiv: https://arxiv.org/abs/2003.03107 · HTML: https://mlsanity.com/paper/2003.03107/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Mar 2020.

## Built on by (most cited first, 18 of 18)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [NLX-GPT: A Model for Natural Language Explanations in Vision and Vision-Language Tasks](https://mlsanity.com/paper/2203.05081/) (2022)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)
- [Model-Agnostic Gender Debiased Image Captioning](https://mlsanity.com/paper/2304.03693/) (2023)
- [Journalistic Guidelines Aware News Image Captioning](https://mlsanity.com/paper/2109.02865/) (2021)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [Simple Token-Level Confidence Improves Caption Correctness](https://mlsanity.com/paper/2305.07021/) (2023)
- [Vision-Language Navigation with Embodied Intelligence: A Survey](https://mlsanity.com/paper/2402.14304/) (2024)
- [Image Captioning via Compact Bidirectional Architecture](https://mlsanity.com/paper/2201.01984/) (2022)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [Explicit Image Caption Editing](https://mlsanity.com/paper/2207.09625/) (2022)
- [Efficient Modeling of Future Context for Image Captioning](https://mlsanity.com/paper/2207.10897/) (2022)
- [Bidirectional Awareness Induction in Autoregressive Seq2Seq Models](https://mlsanity.com/paper/2408.13959/) (2024)
- [Cross Modification Attention Based Deliberation Model for Image Captioning](https://mlsanity.com/paper/2109.08411/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Controllable Image Captioning](https://mlsanity.com/paper/2204.13324/) (2022)
- [Semi-Supervised Image Captioning Considering Wasserstein Graph Matching](https://mlsanity.com/paper/2403.17995/) (2024)

## Abstract (arXiv)

Most image captioning frameworks generate captions directly from images, learning a mapping from visual features to natural language. However, editing existing captions can be easier than generating new ones from scratch. Intuitively, when editing captions, a model is not required to learn information that is already present in the caption (i.e. sentence structure), enabling it to focus on fixing details (e.g. replacing repetitive words). This paper proposes a novel approach to image captioning based on iterative adaptive refinement of an existing caption. Specifically, our caption-editing model consisting of two sub-modules: (1) EditNet, a language module with an adaptive copy mechanism (Copy-LSTM) and a Selective Copy Memory Attention mechanism (SCMA), and (2) DCNet, an LSTM-based denoising auto-encoder. These components enable our model to directly copy from and modify existing captions. Experiments demonstrate that our new approach achieves state-of-art performance on the MS COCO dataset both with and without sequence-level training.