mlsanityData to 7 Oct 2026

Vista-LLaMA: Reducing Hallucination in Video Language Models via Equal Distance to Visual Tokens

arXiv 2312.08870 ↗First posted 12 Dec 2023cs.CVcited by 70

Not on the map. Why?

Cited by 32 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in large video-language models have displayed promising outcomes in video comprehension. Current approaches straightforwardly convert video into language tokens and employ large language models for multi-modal tasks. However, this method often leads to the generation of irrelevant content, commonly known as "hallucination", as the length of the text increases and the impact of the video diminishes. To address this problem, we propose Vista-LLaMA, a novel framework that maintains the consistent distance between all visual tokens and any language tokens, irrespective of the generated text length. Vista-LLaMA omits relative position encoding when determining attention weights between visual and text tokens, retaining the position encoding for text and text tokens. This amplifies the effect of visual tokens on text generation, especially when the relative distance is longer between visual and text tokens. The proposed attention mechanism significantly reduces the chance of producing irrelevant text related to the video content. Furthermore, we present a sequential visual projector that projects the current video frame into tokens of language space with the assistance of the previous frame. This approach not only captures the temporal relationship within the video, but also allows less visual tokens to encompass the entire video. Our approach significantly outperforms various previous methods (e.g., Video-ChatGPT, MovieChat) on four challenging open-ended video question answering benchmarks. We reach an accuracy of 60.7 on the zero-shot NExT-QA and 60.5 on the zero-shot MSRVTT-QA, setting a new state-of-the-art performance. This project is available at https://jinxxian.github.io/Vista-LLaMA.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

31 of 43 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
19 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  2. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  3. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  6. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  7. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  8. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  9. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  13. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  14. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  15. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  16. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  17. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  18. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  19. 2023Temporal Perceiving Video-Language Pre-trainingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 29; 2026: 13.

70 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  3. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  4. 2023Video Understanding with Large Language Models: A Surveycited by 203
  5. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  6. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  7. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  8. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  9. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  10. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  11. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  12. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
38 more
  1. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 78
  2. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  3. 2024CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarioscited by 53
  4. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  5. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  6. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 42
  7. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  8. 2024Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrievalcited by 36
  9. 2024HeadStudio: Text to Animatable Head Avatars with 3D Gaussian Splattingcited by 33
  10. 2024From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understandingcited by 29
  11. 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
  12. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  13. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  14. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  15. 2024Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMscited by 17
  16. 2025PaMi-VDPO: Mitigating Video Hallucinations by Prompt-Aware Multi-Instance Video Preference Learningcited by 15
  17. 2024Efficient Temporal Extrapolation of Multimodal Large Language Models with Temporal Grounding Bridgecited by 14
  18. 2025Dr.V: A Hierarchical Perception-Temporal-Cognition Framework to Diagnose Video Hallucination by Fine-grained Spatial-Temporal Groundingcited by 13
  19. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMscited by 11
  20. 2026Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decodingcited by 11
  21. 2024Hierarchical Memory for Long Video QAcited by 9
  22. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Modelscited by 9
  23. 2025MVU-Eval: Towards Multi-Video Understanding Evaluation for Multimodal LLMscited by 9
  24. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  25. 2024VideoSAVi: Self-Aligned Video Language Models without Human Supervisioncited by 7
  26. 2025Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineeringcited by 7
  27. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigationcited by 7
  28. 2025MT-Video-Bench: A Holistic Video Understanding Benchmark for Evaluating Multimodal LLMs in Multi-Turn Dialoguescited by 7
  29. 2024TC-LLaVA: Rethinking the Transfer from Image to Video Understanding with Temporal Considerationscited by 6
  30. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignmentcited by 6
  31. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMscited by 6
  32. 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
  33. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  34. 2025FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoningcited by 5
  35. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  36. 2025Long-horizon Visual Instruction Generation with Logic and Attribute Self-reflectioncited by 4
  37. 2026CounterVid: Counterfactual Video Generation for Mitigating Action and Temporal Hallucinations in Video-Language Modelscited by 4
  38. 2025Task-Specific Dual-Model Framework for Comprehensive Traffic Safety Video Description and Analysiscited by 3

50 most cited of 70 All 70 (JSON)