# HeroFrame-Bench: Reference-Anchored Evaluation via Rubric--Ranking Co-Evolution for Movie Hero Frame Selection

arXiv: https://arxiv.org/abs/2609.32280 · HTML: https://mlsanity.com/paper/2609.32280/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 26 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [MLVU: Benchmarking Multi-task Long Video Understanding](https://mlsanity.com/paper/2406.04264/) (2024)
- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [Movie Description](https://mlsanity.com/paper/1605.03705/) (2016)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [GenAI Arena: An Open Evaluation Platform for Generative Models](https://mlsanity.com/paper/2406.04485/) (2024)
- [MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation](https://mlsanity.com/paper/2411.15262/) (2024)
- [SkillGen: Verified Inference-Time Agent Skill Synthesis](https://mlsanity.com/paper/2605.10999/) (2026)
- [To Click or Not To Click: Automatic Selection of Beautiful Thumbnails from Videos](https://mlsanity.com/paper/1609.01388/) (2016)
- [SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.14777/) (2026)
- [Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding](https://mlsanity.com/paper/2603.00512/) (2026)
- [Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method](https://mlsanity.com/paper/2305.20042/) (2023)
- [KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding](https://mlsanity.com/paper/2512.14017/) (2025)
- [Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?](https://mlsanity.com/paper/2605.12684/) (2026)

## Abstract (arXiv)

Hero frames are in-film stills used as source imagery for theatrical posters, streaming cover art, film database listings, and other promotional placements. As the first visual entry point, they shape audiences' initial impressions of the movie and their subsequent willingness to watch it. Selecting these frames, a task we term hero frame selection, requires balancing content relevance with aesthetic appeal. A related task is keyframe selection, yet its benchmarks prioritize relevance over aesthetics, using either finite annotations that exclude valid alternatives or VideoQA that entangles selection quality with downstream model capability. We therefore introduce HeroFrame-Bench, built through a scalable VLM-as-a-Judge framework. We construct multimodal contexts from diverse metadata to ground a VLM judge that scores selected frames using our Reference-anchored Percentile. The percentile is obtained by inserting each frame into reusable, pre-ranked reference chains, enabling direct, extensible, and reliable evaluation. To reduce ambiguity and improve consistency in these subjective judgements, we further propose Rubric-Ranking Co-Evolution, which generates movie-specific rubrics to condition the VLM judge and refines rubrics jointly with the resulting rankings. Within this process, we introduce several verifiable signals, most notably the Inverted Rubric Attack, to select robust rubrics. Finally, HeroFrame-Bench is instantiated over 204 movies with 2,031 reference chains and 1,970 learned rubrics. We build an annotation interface for human-alignment studies which show that our construction design improves VLM agreement with human from 77.56% to 83.78%. Evaluation on multiple methods show that hero frame selection remains challenging.