# simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions

arXiv: https://arxiv.org/abs/1808.08732 · HTML: https://mlsanity.com/paper/1808.08732/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2018.

## Built on by (most cited first, 20 of 20)

- [Exploring and Distilling Posterior and Prior Knowledge for Radiology Report Generation](https://mlsanity.com/paper/2106.06963/) (2021)
- [Competence-based Multimodal Curriculum Learning for Medical Report Generation](https://mlsanity.com/paper/2206.14579/) (2022)
- [Contrastive Attention for Automatic Chest X-ray Report Generation](https://mlsanity.com/paper/2106.06965/) (2021)
- [AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation](https://mlsanity.com/paper/2203.10095/) (2022)
- [Cross-modal Prototype Driven Network for Radiology Report Generation](https://mlsanity.com/paper/2207.04818/) (2022)
- [Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation](https://mlsanity.com/paper/2111.04318/) (2021)
- [Rethinking Skip Connection with Layer Normalization in Transformers and ResNets](https://mlsanity.com/paper/2105.07205/) (2021)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Memory-based Cross-modal Semantic Alignment Network for Radiology Report Generation](https://mlsanity.com/paper/2404.00588/) (2024)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [Exploring Semantic Relationships for Unpaired Image Captioning](https://mlsanity.com/paper/2106.10658/) (2021)
- [Graph-in-Graph Network for Automatic Gene Ontology Description Generation](https://mlsanity.com/paper/2206.05311/) (2022)
- [DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention](https://mlsanity.com/paper/2210.16431/) (2022)
- [Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling](https://mlsanity.com/paper/2203.05346/) (2022)
- [Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model](https://mlsanity.com/paper/2107.01571/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [An Emergency Disposal Decision-making Method with Human--Machine Collaboration](https://mlsanity.com/paper/2305.18165/) (2023)
- [IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer](https://mlsanity.com/paper/2308.05633/) (2023)
- [LLaMA-XR: A Novel Framework for Radiology Report Generation using LLaMA and QLoRA Fine Tuning](https://mlsanity.com/paper/2506.03178/) (2025)
- [EIR: Enhanced Image Representations for Medical Report Generation](https://mlsanity.com/paper/2512.23185/) (2025)

## Abstract (arXiv)

The encode-decoder framework has shown recent success in image captioning. Visual attention, which is good at detailedness, and semantic attention, which is good at comprehensiveness, have been separately proposed to ground the caption on the image. In this paper, we propose the Stepwise Image-Topic Merging Network (simNet) that makes use of the two kinds of attention at the same time. At each time step when generating the caption, the decoder adaptively merges the attentive information in the extracted topics and the image according to the generated context, so that the visual information and the semantic information can be effectively combined. The proposed approach is evaluated on two benchmark datasets and reaches the state-of-the-art performances.(The code is available at https://github.com/lancopku/simNet)