# ContCap: A scalable framework for continual image captioning

arXiv: https://arxiv.org/abs/1909.08745 · HTML: https://mlsanity.com/paper/1909.08745/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Sep 2019.

## Built on by (most cited first, 11 of 11)

- [Recent Advances of Continual Learning in Computer Vision: An Overview](https://mlsanity.com/paper/2109.11369/) (2021)
- [How to Teach Large Multimodal Models New Skills](https://mlsanity.com/paper/2510.08564/) (2025)
- [Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation](https://mlsanity.com/paper/2406.19297/) (2024)
- [Task Formulation Matters When Learning Continually: A Case Study in Visual Question Answering](https://mlsanity.com/paper/2210.00044/) (2022)
- [Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images](https://mlsanity.com/paper/2407.14242/) (2024)
- [Towards Adaptable and Interactive Image Captioning with Data Augmentation and Episodic Memory](https://mlsanity.com/paper/2306.03500/) (2023)
- [Overcoming Catastrophic Forgetting by XAI](https://mlsanity.com/paper/2211.14177/) (2022)
- [Semi-supervised Neural Machine Translation with Consistency Regularization for Low-Resource Languages](https://mlsanity.com/paper/2304.00557/) (2023)
- [Putting Humans in the Image Captioning Loop](https://mlsanity.com/paper/2306.03476/) (2023)
- [Continual Learning for Image Captioning through Improved Image-Text Alignment](https://mlsanity.com/paper/2510.06009/) (2025)
- [Evolving Without Ending: Unifying Multimodal Incremental Learning for Continual Panoptic Perception](https://mlsanity.com/paper/2601.15643/) (2026)

## Abstract (arXiv)

While advanced image captioning systems are increasingly describing images coherently and exactly, recent progress in continual learning allows deep learning models to avoid catastrophic forgetting. However, the domain where image captioning working with continual learning has not yet been explored. We define the task in which we consolidate continual learning and image captioning as continual image captioning. In this work, we propose ContCap, a framework generating captions over a series of new tasks coming, seamlessly integrating continual learning into image captioning besides addressing catastrophic forgetting. After proving forgetting in image captioning, we propose various techniques to overcome the forgetting dilemma by taking a simple fine-tuning schema as the baseline. We split MS-COCO 2014 dataset to perform experiments in class-incremental settings without revisiting dataset of previously provided tasks. Experiments show remarkable improvements in the performance on the old tasks while the figures for the new surprisingly surpass fine-tuning. Our framework also offers a scalable solution for continual image or video captioning.