mlsanityData to 7 Oct 2026

AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark

arXiv 2410.03051 ↗First posted 4 Oct 2024cs.CVcited by 137

Not on the map. Why?

Cited by 51 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Builds on 141 of its 168 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

141 of 168 references.

References · 141

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  13. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  14. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  15. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  16. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  17. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  18. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  19. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  20. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  21. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  22. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  23. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  24. 2022Token Merging: Your ViT But Fastercited by 995
  25. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  26. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  27. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  28. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  29. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  30. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  31. 2017Dense-Captioning Events in Videoscited by 789
  32. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  33. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  34. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  35. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  36. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  37. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  38. 2023VILA: On Pre-training for Visual Language Modelscited by 686

50 most cited of 141 All 141 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 76; 2026: 47.

137 papers.

Sort:

By area:

  1. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  3. 2024SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memorycited by 91
  4. 2025TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videoscited by 90
  5. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  6. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  7. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 63
  8. 2025Towards Understanding Camera Motions in Any Videocited by 62
  9. 2025VL-JEPA: Joint Embedding Predictive Architecture for Vision-languagecited by 61
  10. 2024FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Modelscited by 52
  11. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  12. 2024Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Accelerationcited by 40
38 more
  1. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 40
  2. 2025video-SALMONN 2: Caption-Enhanced Audio-Visual Large Language Modelscited by 39
  3. 2024Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Accelerationcited by 38
  4. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  5. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  6. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 35
  7. 2025JarvisIR: Elevating Autonomous Driving Perception with Intelligent Image Restorationcited by 32
  8. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  9. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  10. 2025AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestrationcited by 26
  11. 2025Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perceptioncited by 25
  12. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 24
  13. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 23
  14. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  15. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  16. 2025MASH-VLM: Mitigating Action-Scene Hallucination in Video-LLMs through Disentangled Spatial-Temporal Representationscited by 22
  17. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  18. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 22
  19. 2025VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimizationcited by 21
  20. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  21. 2024VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?cited by 18
  22. 2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 18
  23. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  24. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17
  25. 2025Vid-SME: Membership Inference Attacks against Large Video Understanding Modelscited by 17
  26. 2025UGC-VideoCaptioner: An Omni UGC Video Detail Caption Model and New Benchmarkscited by 17
  27. 2024p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decaycited by 16
  28. 2025ARGUS: Hallucination and Omission Evaluation in Video-LLMscited by 16
  29. 2025Factuality Matters: When Image Generation and Editing Meet Structured Visualscited by 16
  30. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  31. 2025Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methodscited by 15
  32. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  33. 2025ShortV: Efficient Multimodal Large Language Models by Freezing Visual Tokens in Ineffective Layerscited by 13
  34. 2025TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Actioncited by 13
  35. 2025PICABench: How Far Are We from Physically Realistic Image Editing?cited by 13
  36. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  37. 2026DiaDem: Advancing Dialogue Descriptions in Audiovisual Video Captioning for Multimodal Large Language Modelscited by 12
  38. 2024MVTamperBench: Evaluating Robustness of Vision-Language Modelscited by 11

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
  2. 2025Caption Anything in Video: Fine-grained Object-centric Captioning via Spatiotemporal Multimodal Prompting
  3. 2025Evaluating Multimodal Large Language Models on Video Captioning via Monte Carlo Tree Search
  4. 2025LongCaptioning: Unlocking the Power of Long Video Caption Generation in Large Multimodal Models
  5. 2024Vript: A Video Is Worth Thousands of Words
  6. 2026Advancing Video-Text Pretraining with Multi-View Captions
  7. 2026VIVECaption: A Split Approach to Caption Quality Improvement
  8. 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis