# REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation

arXiv: https://arxiv.org/abs/2606.20736 · HTML: https://mlsanity.com/paper/2606.20736/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 15 of 15)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluation](https://mlsanity.com/paper/2502.17521/) (2025)
- [Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts](https://mlsanity.com/paper/2511.04655/) (2025)
- [Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping](https://mlsanity.com/paper/2410.08695/) (2024)
- [AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?](https://mlsanity.com/paper/2410.21259/) (2024)
- [LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content](https://mlsanity.com/paper/2410.10783/) (2024)
- [Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLM](https://mlsanity.com/paper/2411.03823/) (2024)
- [VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes](https://mlsanity.com/paper/2509.25339/) (2025)
- [MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge](https://mlsanity.com/paper/2604.18164/) (2026)
- [HRScene: How Far Are VLMs from Effective High-Resolution Image Understanding?](https://mlsanity.com/paper/2504.18406/) (2025)
- [KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution](https://mlsanity.com/paper/2510.21182/) (2025)

## Abstract (arXiv)

Static visual question answering (VQA) benchmarks age quickly: Once the items leak into training corpora, scores can reflect memorization rather than genuine visual ability, thus obscuring real progress. Rebuilding high-quality benchmarks such as V*Bench requires substantial human annotation, yet each static release can quickly become another leaked artifact. We propose ReKey, a live benchmark protocol that randomly regenerates the answer-bearing local detail, or visual key, in real images at evaluation time. Using human-validated edit slots, ReKey samples fresh instances with new answers, construction-grounded labels, and controlled visual-search difficulty. On V*Bench, the ReKey regenerated benchmark reveals a sharp score jump across eight frontier vision-language models (VLMs): The original items score 9.5--18.8 percentage points higher than the regenerated variants. By making the visual key renewable, ReKey keeps evaluation fresh as models and training data evolve.