mlsanityData to 7 Oct 2026

VEGAS: Human-Aligned Video Caption Evaluation via Gaze

arXiv 2607.08489 ↗First posted 9 Jul 2026cs.CV cs.AI cs.HCcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers' attention. We propose VEGAS (Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus. To evaluate VEGAS, we curate a dataset of egocentric activities and instructional slides paired with synchronized gaze and reference annotations. We then select captions based on VEGAS via rejection sampling without model retraining. Experiments show that VEGAS-selected captions align significantly better with human focus and improve downstream caption-to-video retrieval, demonstrating the practical utility of incorporating viewer attention during inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 29 references that are in the map. First posted on arXiv 9 Jul 2026.

Builds on

19 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models18 Feb

Other references · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  4. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  5. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  6. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
  7. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  8. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  9. 2020In the Eye of the Beholder: Gaze and Actions in First Person Videocited by 68
  10. 2013Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognitioncited by 36
  11. 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Promptingcited by 21
  12. 2016Seeing with Humans: Gaze-Assisted Neural Image Captioningcited by 20
6 more
  1. 2017Supervising Neural Attention Models for Video Captioning by Human Gaze Datacited by 18
  2. 2025SkillSight: Efficient First-Person Skill Assessment with Gazecited by 4
  3. 2025VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DRcited by 3
  4. 2026Controlling Reading Ease with Gaze-Guided Text Generationcited by 3
  5. 2025Eyes on Target: Gaze-Aware Object Detection in Egocentric Videocited by 2
  6. 2026ViSIL: Unified Evaluation of Information Loss in Multimodal Video Captioningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning
  2. 2026Learning to Rank Caption Chains for Video-Text Alignment
  3. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughness
  4. 2024Distilling Vision-Language Models on Millions of Videos
  5. 2026Building a Precise Video Language with Human-AI Oversight
  6. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
  7. 2025VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
  8. 2026ProCap: Prominence-guided Object Rectification for Faithful and Comprehensive Video Captioning