mlsanityData to 7 Oct 2026

HeroFrame-Bench: Reference-Anchored Evaluation via Rubric--Ranking Co-Evolution for Movie Hero Frame Selection

arXiv 2609.32280 ↗First posted 26 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Hero frames are in-film stills used as source imagery for theatrical posters, streaming cover art, film database listings, and other promotional placements. As the first visual entry point, they shape audiences' initial impressions of the movie and their subsequent willingness to watch it. Selecting these frames, a task we term hero frame selection, requires balancing content relevance with aesthetic appeal. A related task is keyframe selection, yet its benchmarks prioritize relevance over aesthetics, using either finite annotations that exclude valid alternatives or VideoQA that entangles selection quality with downstream model capability. We therefore introduce HeroFrame-Bench, built through a scalable VLM-as-a-Judge framework. We construct multimodal contexts from diverse metadata to ground a VLM judge that scores selected frames using our Reference-anchored Percentile. The percentile is obtained by inserting each frame into reusable, pre-ranked reference chains, enabling direct, extensible, and reliable evaluation. To reduce ambiguity and improve consistency in these subjective judgements, we further propose Rubric-Ranking Co-Evolution, which generates movie-specific rubrics to condition the VLM judge and refines rubrics jointly with the resulting rankings. Within this process, we introduce several verifiable signals, most notably the Inverted Rubric Attack, to select robust rubrics. Finally, HeroFrame-Bench is instantiated over 204 movies with 2,031 reference chains and 1,970 learned rubrics. We build an annotation interface for human-alignment studies which show that our construction design improves VLM agreement with human from 77.56% to 83.78%. Evaluation on multiple methods show that hero frame selection remains challenging.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

25 of 43 references.

References · 25

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  7. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  8. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  9. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  12. 2024MLVU: Benchmarking Multi-task Long Video Understandingcited by 275
13 more
  1. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 196
  2. 2016Movie Descriptioncited by 180
  3. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  4. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 81
  5. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 81
  6. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  7. 2016To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videoscited by 33
  8. 2026SkillGen: Verified Inference-Time Agent Skill Synthesiscited by 29
  9. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 23
  10. 2026Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understandingcited by 11
  11. 2023Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote methodcited by 9
  12. 2025KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understandingcited by 6
  13. 2026Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?cited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026AesRM: Improving Video Aesthetics with Expert-Level Feedback
  2. 2026Popcorn: A Configurable Benchmark for Visual Evidence in Multimodal Movie Recommendation ↗
  3. 2026EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation
  4. 2026VGA-BenchV2: An Expanded Unified Benchmark and Multi-Model Framework for Evaluating Video Aesthetics and Generation Quality
  5. 2026VideoArgus: Agentic Rubric-Grounded Unified Evaluation for Video Generation and Editing
  6. 2026SenBen: Sensitive Scene Graphs for Explainable Content Moderation
  7. 2026Evaluating Multimodal Narrative Understanding of Popular Hollywood Films
  8. 2025CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation