# Regularizing RNNs for Caption Generation by Reconstructing The Past with The Present

arXiv: https://arxiv.org/abs/1803.11439 · HTML: https://mlsanity.com/paper/1803.11439/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2018.

## Built on by (most cited first, 15 of 15)

- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [M^4I: Multi-modal Models Membership Inference](https://mlsanity.com/paper/2209.06997/) (2022)
- [MMNet: Multi-Collaboration and Multi-Supervision Network for Sequential Deepfake Detection](https://mlsanity.com/paper/2307.02733/) (2023)
- [Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention](https://mlsanity.com/paper/2109.02955/) (2021)
- [Reverse Stable Diffusion: What prompt was used to generate this image?](https://mlsanity.com/paper/2308.01472/) (2023)
- [Self-Supervised Image Captioning with CLIP](https://mlsanity.com/paper/2306.15111/) (2023)
- [Macroscopic Control of Text Generation for Image Captioning](https://mlsanity.com/paper/2101.08000/) (2021)
- [Visual-aware Attention Dual-stream Decoder for Video Captioning](https://mlsanity.com/paper/2110.08578/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [Dense Video Captioning using Graph-based Sentence Summarization](https://mlsanity.com/paper/2506.20583/) (2025)
- [Dual Branch Network Towards Accurate Printed Mathematical Expression Recognition](https://mlsanity.com/paper/2312.09030/) (2023)
- [Show, Tell and Summarize: Dense Video Captioning Using Visual Cue Aided Sentence Summarization](https://mlsanity.com/paper/2506.20567/) (2025)

## Abstract (arXiv)

Recently, caption generation with an encoder-decoder framework has been extensively studied and applied in different domains, such as image captioning, code captioning, and so on. In this paper, we propose a novel architecture, namely Auto-Reconstructor Network (ARNet), which, coupling with the conventional encoder-decoder framework, works in an end-to-end fashion to generate captions. ARNet aims at reconstructing the previous hidden state with the present one, besides behaving as the input-dependent transition operator. Therefore, ARNet encourages the current hidden state to embed more information from the previous one, which can help regularize the transition dynamics of recurrent neural networks (RNNs). Extensive experimental results show that our proposed ARNet boosts the performance over the existing encoder-decoder models on both image captioning and source code captioning tasks. Additionally, ARNet remarkably reduces the discrepancy between training and inference processes for caption generation. Furthermore, the performance on permuted sequential MNIST demonstrates that ARNet can effectively regularize RNN, especially on modeling long-term dependencies. Our code is available at: https://github.com/chenxinpeng/ARNet