# MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

arXiv: https://arxiv.org/abs/2610.18461 · HTML: https://mlsanity.com/paper/2610.18461/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Image Quality Assessment: Unifying Structure and Texture Similarity](https://mlsanity.com/paper/2004.07728/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation](https://mlsanity.com/paper/2407.02371/) (2024)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [Learning Blind Video Temporal Consistency](https://mlsanity.com/paper/1808.00449/) (2018)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [Towards An End-to-End Framework for Flow-Guided Video Inpainting](https://mlsanity.com/paper/2204.02663/) (2022)
- [Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN](https://mlsanity.com/paper/1904.10247/) (2019)
- [AVID: Any-Length Video Inpainting with Diffusion Model](https://mlsanity.com/paper/2312.03816/) (2023)
- [DiffuEraser: A Diffusion Model for Video Inpainting](https://mlsanity.com/paper/2501.10018/) (2025)
- [The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English](https://mlsanity.com/paper/1902.01382/) (2019)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [ROSE: Remove Objects with Side Effects in Videos](https://mlsanity.com/paper/2508.18633/) (2025)
- [OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2503.18033/) (2025)
- [EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing](https://mlsanity.com/paper/2603.19224/) (2026)
- [The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting](https://mlsanity.com/paper/2105.05332/) (2021)
- [ViTEraser: Harnessing the Power of Vision Transformers for Scene Text Removal with SegMIM Pretraining](https://mlsanity.com/paper/2306.12106/) (2023)
- [Elevating Flow-Guided Video Inpainting with Reference Generation](https://mlsanity.com/paper/2412.08975/) (2024)
- [Deep Blind Video Decaptioning by Temporal Aggregation and Recurrence](https://mlsanity.com/paper/1905.02949/) (2019)
- [BVINet: Unlocking Blind Video Inpainting with Zero Annotations](https://mlsanity.com/paper/2502.01181/) (2025)
- [CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal](https://mlsanity.com/paper/2603.21901/) (2026)
- [PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media](https://mlsanity.com/paper/2605.14534/) (2026)
- [Leveraging Text Localization for Scene Text Removal via Text-aware Masked Image Modeling](https://mlsanity.com/paper/2409.13431/) (2024)

## Abstract (arXiv)

Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test sets, and generic video-inpainting protocols that fail to adequately capture localized restoration errors, unintended modifications, temporal instability, and incomplete outputs. In this paper, we introduce MFVSR, a large-scale multi-style benchmark for fine-grained and reliability-aware video subtitle removal evaluation. MFVSR distinguishes itself through three key features: (1) Large-scale and diverse data, comprising 3,000 paired videos from Real-HQ, Real-LQ, and AI-Generated sources, with an auxiliary set of 953 in-the-wild real videos for real-world coverage analysis; (2) Controlled subtitle diversity, spanning 16 languages and four factors covering spatial layout, temporal behavior, visual style, and language mode; (3) Fine-grained evaluation, introducing an information-conditioned, spatially decoupled protocol that separates blind from mask-guided removal and measures Restoration Fidelity, Preservation Fidelity, Temporal Fidelity, coverage, and failures. Experiments on six representative methods reveal substantial discrepancies between local restoration, outside-region preservation, temporal consistency, and full-set reliability that are obscured by conventional full-frame metrics. MFVSR therefore provides a more comprehensive and diagnostic foundation for evaluating and advancing video subtitle removal methods.