# Recurrent Fusion Network for Image Captioning

arXiv: https://arxiv.org/abs/1807.09986 · HTML: https://mlsanity.com/paper/1807.09986/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2018.

## Built on by (most cited first, 30 of 43)

- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Context-aware Biaffine Localizing Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2103.11555/) (2021)
- [D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2112.01551/) (2021)
- [Memory-Guided Semantic Learning Network for Temporal Sentence Grounding](https://mlsanity.com/paper/2201.00454/) (2022)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [End-to-End Transformer Based Model for Image Captioning](https://mlsanity.com/paper/2203.15350/) (2022)
- [Semantic-Conditional Diffusion Networks for Image Captioning](https://mlsanity.com/paper/2212.03099/) (2022)
- [Exploring Optical-Flow-Guided Motion and Detection-Based Appearance for Temporal Sentence Grounding](https://mlsanity.com/paper/2203.02966/) (2022)
- [Reducing the Vision and Language Bias for Temporal Sentence Grounding](https://mlsanity.com/paper/2207.13457/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [Towards Local Visual Modeling for Image Captioning](https://mlsanity.com/paper/2302.06098/) (2023)
- [Beyond a Pre-Trained Object Detector: Cross-Modal Textual and Visual Context for Image Captioning](https://mlsanity.com/paper/2205.04363/) (2022)
- [Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation](https://mlsanity.com/paper/2303.15932/) (2023)
- [Auto-Parsing Network for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2108.10568/) (2021)
- [Neural Attention for Image Captioning: Review of Outstanding Methods](https://mlsanity.com/paper/2111.15015/) (2021)
- [HAAV: Hierarchical Aggregation of Augmented Views for Image Captioning](https://mlsanity.com/paper/2305.16295/) (2023)
- [Image Captioning via Compact Bidirectional Architecture](https://mlsanity.com/paper/2201.01984/) (2022)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [Geometry Attention Transformer with Position-aware LSTMs for Image Captioning](https://mlsanity.com/paper/2110.00335/) (2021)
- [DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention](https://mlsanity.com/paper/2210.16431/) (2022)
- [Top-Down Framework for Weakly-supervised Grounded Image Captioning](https://mlsanity.com/paper/2306.07490/) (2023)
- [Label-Attention Transformer with Geometrically Coherent Objects for Image Captioning](https://mlsanity.com/paper/2109.07799/) (2021)
- [Image Captioning In the Transformer Age](https://mlsanity.com/paper/2204.07374/) (2022)
- [Transform-Equivariant Consistency Learning for Temporal Sentence Grounding](https://mlsanity.com/paper/2305.04123/) (2023)
- [Image Captioning via Dynamic Path Customization](https://mlsanity.com/paper/2406.00334/) (2024)
- [Guiding Attention using Partial-Order Relationships for Image Captioning](https://mlsanity.com/paper/2204.07476/) (2022)
- [Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning](https://mlsanity.com/paper/2210.01338/) (2022)

## Abstract (arXiv)

Recently, much advance has been made in image captioning, and an encoder-decoder framework has been adopted by all the state-of-the-art models. Under this framework, an input image is encoded by a convolutional neural network (CNN) and then translated into natural language with a recurrent neural network (RNN). The existing models counting on this framework merely employ one kind of CNNs, e.g., ResNet or Inception-X, which describe image contents from only one specific view point. Thus, the semantic meaning of an input image cannot be comprehensively understood, which restricts the performance of captioning. In this paper, in order to exploit the complementary information from multiple encoders, we propose a novel Recurrent Fusion Network (RFNet) for tackling image captioning. The fusion process in our model can exploit the interactions among the outputs of the image encoders and then generate new compact yet informative representations for the decoder. Experiments on the MSCOCO dataset demonstrate the effectiveness of our proposed RFNet, which sets a new state-of-the-art for image captioning.