# Fine-Grained Captioning of Long Videos through Scene Graph Consolidation

arXiv: https://arxiv.org/abs/2502.16427 · HTML: https://mlsanity.com/paper/2502.16427/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 23 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [MovieChat: From Dense Token to Sparse Memory for Long Video Understanding](https://mlsanity.com/paper/2307.16449/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning](https://mlsanity.com/paper/2404.16994/) (2024)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [VTimeLLM: Empower LLM to Grasp Video Moments](https://mlsanity.com/paper/2311.18445/) (2023)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [LongVLM: Efficient Long Video Understanding via Large Language Models](https://mlsanity.com/paper/2404.03384/) (2024)
- [OmniVL:One Foundation Model for Image-Language and Video-Language Tasks](https://mlsanity.com/paper/2209.07526/) (2022)
- [AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark](https://mlsanity.com/paper/2410.03051/) (2024)
- [Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners](https://mlsanity.com/paper/2205.10747/) (2022)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)

## Built on by (most cited first, 8 of 8)

- [Watch, Remember, Reason: Human-View Video Understanding with MLLMs](https://mlsanity.com/paper/2606.07433/) (2026)
- [IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory](https://mlsanity.com/paper/2604.20136/) (2026)
- [Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding](https://mlsanity.com/paper/2605.07575/) (2026)
- [CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales](https://mlsanity.com/paper/2606.21949/) (2026)
- [Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs](https://mlsanity.com/paper/2606.25842/) (2026)
- [Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs](https://mlsanity.com/paper/2607.02425/) (2026)
- [DynaStride: Dynamic Stride Windowing with MMCoT for Instructional Multi-Scene Captioning](https://mlsanity.com/paper/2510.23907/) (2025)
- [PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary](https://mlsanity.com/paper/2610.16066/) (2026)

## Abstract (arXiv)

Recent advances in vision-language models have led to impressive progress in caption generation for images and short video clips. However, these models remain constrained by their limited temporal receptive fields, making it difficult to produce coherent and comprehensive captions for long videos. While several methods have been proposed to aggregate information across video segments, they often rely on supervised fine-tuning or incur significant computational overhead. To address these challenges, we introduce a novel framework for long video captioning based on graph consolidation. Our approach first generates segment-level captions, corresponding to individual frames or short video intervals, using off-the-shelf visual captioning models. These captions are then parsed into individual scene graphs, which are subsequently consolidated into a unified graph representation that preserves both holistic context and fine-grained details throughout the video. A lightweight graph-to-text decoder then produces the final video-level caption. This framework effectively extends the temporal understanding capabilities of existing models without requiring any additional fine-tuning on long video datasets. Experimental results show that our method significantly outperforms existing LLM-based consolidation approaches, achieving strong zero-shot performance while substantially reducing computational costs.