mlsanityData to 7 Oct 2026

Video ReCap: Recursive Captioning of Hour-Long Videos

arXiv 2402.13250 ↗First posted 20 Feb 2024cs.CVcited by 88

Not on the map. Why?

Cited by 20 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Most video captioning models are designed to process short video clips of few seconds and output text describing low-level visual concepts (e.g., objects, scenes, atomic actions). However, most real-world videos last for minutes or hours and have a complex hierarchical structure spanning different temporal granularities. We propose Video ReCap, a recursive video captioning model that can process video inputs of dramatically different lengths (from 1 second to 2 hours) and output video captions at multiple hierarchy levels. The recursive video-language architecture exploits the synergy between different video hierarchies and can process hour-long videos efficiently. We utilize a curriculum learning training scheme to learn the hierarchical structure of videos, starting from clip-level captions describing atomic actions, then focusing on segment-level descriptions, and concluding with generating summaries for hour-long videos. Furthermore, we introduce Ego4D-HCap dataset by augmenting Ego4D with 8,267 manually collected long-range video summaries. Our recursive model can flexibly generate captions at different hierarchy levels while also being useful for other complex video understanding tasks, such as VideoQA on EgoSchema. Data, code, and models are available at: https://sites.google.com/view/vidrecap

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 51 of its 79 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

51 of 79 references.

References · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
38 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  5. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  6. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  7. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  8. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  9. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  10. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  11. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  12. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  13. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  14. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  15. 2022Egocentric Video-Language Pretrainingcited by 231
  16. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  17. 2022Learning Video Representations from Large Language Modelscited by 204
  18. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  19. 2016Movie Descriptioncited by 180
  20. 2015Sequence to Sequence -- Video to Textcited by 157
  21. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  22. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  23. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  24. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  25. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  26. 2018Reconstruction Network for Video Captioningcited by 98
  27. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  28. 2022My View is the Best View: Procedure Learning from Egocentric Videoscited by 81
  29. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  30. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  31. 2017Attention-Based Multimodal Fusion for Video Descriptioncited by 75
  32. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  33. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
  34. 2019Memory-Attended Recurrent Network for Video Captioningcited by 65
  35. 2016Video Captioning with Transferred Semantic Attributescited by 46
  36. 2017Fluency-Guided Cross-Lingual Image Captioningcited by 33
  37. 2015Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioningcited by 30
  38. 2016Hierarchical Boundary-Aware Neural Encoder for Video Captioningcited by 20

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 48; 2026: 18.

88 papers.

Sort:

By area:

  1. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  2. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  3. 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modelingcited by 212
  4. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  5. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  6. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  7. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  8. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  9. 2025Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memorycited by 88
  10. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  11. 2024Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularitycited by 66
  12. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
38 more
  1. 2024Language Repository for Long Video Understandingcited by 55
  2. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  3. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  4. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  5. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  6. 2025Modeling Fine-Grained Hand-Object Dynamics for Egocentric Video Representation Learningcited by 20
  7. 2025Video Summarization with Large Language Modelscited by 16
  8. 2024Do Language Models Understand Time?cited by 14
  9. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 14
  10. 2024StoryTeller: Improving Long Video Description through Global Audio-Visual Character Identificationcited by 13
  11. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  12. 2024Propose, Assess, Search: Harnessing LLMs for Goal-Oriented Planning in Instructional Videoscited by 12
  13. 2025What Is That Talk About? A Video-to-Text Summarization Dataset for Scientific Presentationscited by 11
  14. 2024Progress-Aware Video Frame Captioningcited by 10
  15. 2025ZoomV: Temporal Zoom-in for Efficient Long Video Understandingcited by 10
  16. 2025EGOILLUSION: Benchmarking Hallucinations in Egocentric Video Understandingcited by 10
  17. 2024HCQA @ Ego4D EgoSchema Challenge 2024cited by 9
  18. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  19. 2025VideoMiner: Iteratively Grounding Key Frames of Hour-Long Videos via Tree-based Group Relative Policy Optimizationcited by 9
  20. 2024Encoding and Controlling Global Semantics for Long-form Video Question Answeringcited by 8
  21. 2025Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generationcited by 8
  22. 2025Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Visioncited by 8
  23. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 7
  24. 2025An Egocentric Vision-Language Model based Portable Real-time Smart Assistantcited by 7
  25. 2025ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understandingcited by 7
  26. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  27. 2025HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videoscited by 7
  28. 2024Distractors-Immune Representation Learning with Cross-modal Contrastive Regularization for Change Captioningcited by 6
  29. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  30. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  31. 2025Moment Sampling in Video LLMs for Long-Form Video QAcited by 5
  32. 2024Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experiencecited by 4
  33. 2025From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approachcited by 4
  34. 2025LeAdQA: LLM-Driven Context-Aware Temporal Grounding for Video Question Answeringcited by 4
  35. 2025VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflectioncited by 4
  36. 2024It's Just Another Day: Unique Video Captioning by Discriminative Promptingcited by 3
  37. 2024Extending Video Masked Autoencoders to 128 framescited by 3
  38. 2025DANTE-AD: Dual-Vision Attention Network for Long-Term Audio Descriptioncited by 3

50 most cited of 88 All 88 (JSON)