# HiCM2: Hierarchical Compact Memory Modeling for Dense Video Captioning

arXiv: https://arxiv.org/abs/2412.14585 · HTML: https://mlsanity.com/paper/2412.14585/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 48 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 9 of 9)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2002.06353/) (2020)
- [Translating Videos to Natural Language Using Deep Recurrent Neural Networks](https://mlsanity.com/paper/1412.4729/) (2014)
- [Multimodal Pretraining for Dense Video Captioning](https://mlsanity.com/paper/2011.11760/) (2020)
- [A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformer](https://mlsanity.com/paper/2005.08271/) (2020)
- [Dense-Captioning Events in Videos: SYSU Submission to ActivityNet Challenge 2020](https://mlsanity.com/paper/2006.11693/) (2020)

## Built on by (most cited first, 7 of 7)

- [Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark](https://mlsanity.com/paper/2504.14693/) (2025)
- [AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond](https://mlsanity.com/paper/2509.26636/) (2025)
- [ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QA](https://mlsanity.com/paper/2607.01737/) (2026)
- [SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning](https://mlsanity.com/paper/2603.05437/) (2026)
- [Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning](https://mlsanity.com/paper/2603.11460/) (2026)
- [DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation](https://mlsanity.com/paper/2604.26565/) (2026)
- [Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning](https://mlsanity.com/paper/2607.02963/) (2026)

## Abstract (arXiv)

With the growing demand for solutions to real-world video challenges, interest in dense video captioning (DVC) has been on the rise. DVC involves the automatic captioning and localization of untrimmed videos. Several studies highlight the challenges of DVC and introduce improved methods utilizing prior knowledge, such as pre-training and external memory. In this research, we propose a model that leverages the prior knowledge of human-oriented hierarchical compact memory inspired by human memory hierarchy and cognition. To mimic human-like memory recall, we construct a hierarchical memory and a hierarchical memory reading module. We build an efficient hierarchical compact memory by employing clustering of memory events and summarization using large language models. Comparative experiments demonstrate that this hierarchical memory recall process improves the performance of DVC by achieving state-of-the-art performance on YouCook2 and ViTT datasets.