mlsanityData to 7 Oct 2026

Streaming Dense Video Captioning

arXiv 2404.01297 ↗First posted 1 Apr 2024cs.CVcited by 82

Not on the map. Why?

Abstract (arXiv)

An ideal model for dense video captioning -- predicting captions localized temporally in a video -- should be able to handle long input videos, predict rich, detailed textual descriptions, and be able to produce outputs before processing the entire video. Current state-of-the-art models, however, process a fixed number of downsampled frames, and make a single full prediction after seeing the whole video. We propose a streaming dense video captioning model that consists of two novel components: First, we propose a new memory module, based on clustering incoming tokens, which can handle arbitrarily long videos as the memory is of a fixed size. Second, we develop a streaming decoding algorithm that enables our model to make predictions before the entire video has been processed. Our model achieves this streaming ability, and significantly improves the state-of-the-art on three dense video captioning benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at https://github.com/google-research/scenic.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Builds on 55 of its 67 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

55 of 67 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2021ViViT: A Video Vision Transformercited by 1,610
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
38 more
  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  4. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  5. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  6. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  7. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  8. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 531
  9. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  10. 2019Tracking without bells and whistlescited by 382
  11. 2019Towards Real-Time Multi-Object Trackingcited by 311
  12. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  13. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  14. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  15. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  16. 2022Memorizing Transformerscited by 219
  17. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  18. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  19. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  20. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  21. 2021Towards Long-Form Video Understandingcited by 146
  22. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  23. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  24. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  25. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  26. 2020Multi-modal Dense Video Captioningcited by 113
  27. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  28. 2019The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentationcited by 95
  29. 2020Actor-Context-Actor Relation Network for Spatio-Temporal Action Localizationcited by 91
  30. 2020Multimodal Pretraining for Dense Video Captioningcited by 79
  31. 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 75
  32. 2021Object-Region Video Transformerscited by 73
  33. 2016Online Action Detectioncited by 68
  34. 2018Bidirectional Attentive Fusion with Context Gating for Dense Video Captioningcited by 68
  35. 2022Real-time Online Video Detection with Temporal Smoothing Transformerscited by 58
  36. 2016Online Real-time Multiple Spatiotemporal Action Localisation and Predictioncited by 57
  37. 2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 56
  38. 2023AutoAD: Movie Description in Contextcited by 50

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 44; 2026: 23.

82 papers.

Sort:

By area:

  1. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 216
  2. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  3. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  4. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  5. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  6. 2024MovieChat+: Question-aware Sparse Memory for Long Video Question Answeringcited by 73
  7. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  8. 2025LION-FS: Fast & Slow Video-Language Thinker as Online Video Assistantcited by 56
  9. 2025Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videoscited by 49
  10. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 48
  11. 2025StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognitioncited by 44
  12. 2026A Simple Baseline for Streaming Video Understandingcited by 29
38 more
  1. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  2. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  3. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  4. 2025LiveStar: Live Streaming Assistant for Real-World Online Video Understandingcited by 22
  5. 2024Towards Universal Soccer Video Understandingcited by 21
  6. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  7. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  8. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  9. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  10. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  11. 2025Shot-by-Shot: Film-Grammar-Aware Training-Free Audio Description Generationcited by 13
  12. 2024Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Groundingcited by 12
  13. 2025LongViTU: Instruction Tuning for Long-Form Video Understandingcited by 12
  14. 2025Enrich and Detect: Video Temporal Grounding with Multimodal LLMscited by 12
  15. 2024Progress-Aware Video Frame Captioningcited by 10
  16. 2025Learning Streaming Video Representation via Multitask Trainingcited by 10
  17. 2025MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detectioncited by 10
  18. 2025AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Drivingcited by 10
  19. 2025Chapter-Llama: Efficient Chaptering in Hour-Long Videos with LLMscited by 9
  20. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  21. 2024Wolf: Dense Video Captioning with a World Summarization Frameworkcited by 7
  22. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 7
  23. 2025LiveCC: Learning Video LLM with Streaming Speech Transcription at Scalecited by 7
  24. 2026Speak While Watching: Unleashing TRUE Real-Time Video Understanding Capability of Multimodal Large Language Modelscited by 7
  25. 2025Open-ended Hierarchical Streaming Video Understanding with Vision Language Modelscited by 6
  26. 2025Video Panels for Long Video Understandingcited by 6
  27. 2024FocusChat: Text-guided Long Video Understanding via Spatiotemporal Information Filteringcited by 5
  28. 2025ChronusOmni: Improving Time Awareness of Omni Large Language Modelscited by 5
  29. 2025HuMoCon: Concept Discovery for Human Motion Understandingcited by 4
  30. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  31. 2025VDC-Agent: When Video Detailed Captioners Evolve Themselves via Agentic Self-Reflectioncited by 4
  32. 2025DANTE-AD: Dual-Vision Attention Network for Long-Term Audio Descriptioncited by 3
  33. 2025FocusedAD: Character-centric Movie Audio Descriptioncited by 3
  34. 2025SV3.3B: A Sports Video Understanding Model for Action Recognitioncited by 3
  35. 2025Track-On2: Enhancing Online Point Tracking with Memorycited by 3
  36. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  37. 2025ExpStar: Towards Automatic Commentary Generation for Multi-discipline Scientific Experimentscited by 2
  38. 2025Shot2Tactic-Caption: Multi-Scale Captioning of Badminton Videos for Tactical Understandingcited by 2

50 most cited of 82 All 82 (JSON)