mlsanityData to 14 Oct 2026

ARGUS: Hallucination and Omission Evaluation in Video-LLMs

arXiv 2506.07371 ↗First posted 9 Jun 2025cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Video large language models have not yet been widely deployed, largely due to their tendency to hallucinate. Typical benchmarks for Video-LLMs rely simply on multiple-choice questions. Unfortunately, VideoLLMs hallucinate far more aggressively on freeform text generation tasks like video captioning than they do on multiple choice verification tasks. To address this weakness, we propose ARGUS, a VideoLLM benchmark that measures freeform video captioning performance. By comparing VideoLLM outputs to human ground truth captions, ARGUS quantifies dual metrics. First, we measure the rate of hallucinations in the form of incorrect statements about video content or temporal relationships. Second, we measure the rate at which the model omits important descriptive details. Together, these dual metrics form a comprehensive view of video captioning performance.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 57 of its 69 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

57 of 69 references.

References · 57

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  8. 2024SAM 2: Segment Anything in Images and Videoscited by 2,863
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
38 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  4. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  7. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  10. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  11. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  13. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 410
  14. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  15. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  16. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  17. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  18. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  19. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 140
  20. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  21. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 114
  22. 2020Multi-modal Dense Video Captioningcited by 113
  23. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  24. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 90
  25. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  26. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
  27. 2024Streaming Dense Video Captioningcited by 82
  28. 2020A Better Use of Audio-Visual Cues: Dense Video Captioning with Bi-modal Transformercited by 76
  29. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  30. 2018Jointly Localizing and Describing Events for Dense Video Captioningcited by 60
  31. 2019Streamlined Dense Video Captioningcited by 57
  32. 2024EventHallusion: Diagnosing Event Hallucinations in Video LLMscited by 57
  33. 2025Logical Reasoning in Large Language Models: A Surveycited by 55
  34. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  35. 2024TC-Bench: Benchmarking Temporal Compositionality in Text-to-Video and Image-to-Video Generationcited by 43
  36. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  37. 2024Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrievalcited by 36
  38. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Modelscited by 36

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 11.

16 papers.

Sort:

By area:

  1. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  2. 2025ELV-Halluc: Benchmarking Semantic Aggregation Hallucinations in Long Video Understandingcited by 10
  3. 2025SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapsecited by 9
  4. 2025NOAH: Benchmarking Narrative Prior driven Hallucination and Omission in Video Large Language Modelscited by 2
  5. 2026VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understandingcited by 2
  6. 2025Catalog-Native LLM: Speaking Item-ID Dialect with Less Entanglement for Recommendationcited by 1
  7. 2026Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Modelscited by 1
  8. 2026Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Modelscited by 1
  9. 2026Towards Temporal Compositional Reasoning in Long-Form Sports Videoscited by 1
  10. 2026VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioningcited by 1
  11. 2026Learning to Rank Caption Chains for Video-Text Alignmentcited by 0
  12. 2026Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMscited by 0
4 more
  1. 2026From Priors to Perception: Grounding Video-LLMs in Physical Realitycited by 0
  2. 2026Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Modelscited by 0
  3. 2026Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Modelscited by 0
  4. 2026STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Modelscited by 0