# BeyondMasks: Evaluating Causal and Physical Consistency in Video Object Removal

arXiv: https://arxiv.org/abs/2608.20107 · HTML: https://mlsanity.com/paper/2608.20107/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 54 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Builds on (most cited first, 30 of 35)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [VACE: All-in-One Video Creation and Editing](https://mlsanity.com/paper/2503.07598/) (2025)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models](https://mlsanity.com/paper/2309.15103/) (2023)
- [VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness](https://mlsanity.com/paper/2503.21755/) (2025)
- [YouTube-VOS: Sequence-to-Sequence Video Object Segmentation](https://mlsanity.com/paper/1809.00461/) (2018)
- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [ProPainter: Improving Propagation and Transformer for Video Inpainting](https://mlsanity.com/paper/2309.03897/) (2023)
- [VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation](https://mlsanity.com/paper/2503.06800/) (2025)
- [CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos](https://mlsanity.com/paper/2410.11831/) (2024)
- [InstructVideo: Instructing Video Diffusion Models with Human Feedback](https://mlsanity.com/paper/2312.12490/) (2023)
- [DiffuEraser: A Diffusion Model for Video Inpainting](https://mlsanity.com/paper/2501.10018/) (2025)
- [ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion](https://mlsanity.com/paper/2403.18818/) (2024)
- [Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videos](https://mlsanity.com/paper/2407.16124/) (2024)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibility](https://mlsanity.com/paper/2403.12035/) (2024)
- [MiniMax-Remover: Taming Bad Noise Helps Video Object Removal](https://mlsanity.com/paper/2505.24873/) (2025)
- [InstructVid2Vid: Controllable Video Editing with Natural Language Instructions](https://mlsanity.com/paper/2305.12328/) (2023)
- [VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation](https://mlsanity.com/paper/2503.14350/) (2025)
- [OmniPaint: Mastering Object-Oriented Editing via Disentangled Insertion-Removal Inpainting](https://mlsanity.com/paper/2503.08677/) (2025)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [ROSE: Remove Objects with Side Effects in Videos](https://mlsanity.com/paper/2508.18633/) (2025)
- [OmniEraser: Remove Objects and Their Effects in Images with Paired Video-Frame Data](https://mlsanity.com/paper/2501.07397/) (2025)
- [Dream2Flow: Bridging Video Generation and Open-World Manipulation with 3D Object Flow](https://mlsanity.com/paper/2512.24766/) (2025)
- [SmartEraser: Remove Anything from Images using Masked-Region Guidance](https://mlsanity.com/paper/2501.08279/) (2025)

## Built on by (most cited first, 1 of 1)

- [TripleFlow: Training-Free Video Object Removal by Bridging Residual Editing and Native Generation](https://mlsanity.com/paper/2609.39157/) (2026)

## Abstract (arXiv)

Recent advances in generative video models have significantly improved visual realism in video object removal, yet evaluation protocols still focus on masked region fidelity, treating removal as local inpainting. In real scenes, object removal is a causal intervention: eliminating an object also requires removing its induced physical effects, such as shadows, reflections, illumination changes, translucency, and dynamic traces. Existing benchmarks lack aligned clean references or remain limited to simplified synthetic settings, preventing systematic evaluation of causal consistency. We introduce BeyondMasks, a paired benchmark for causally consistent video object removal, consisting of temporally aligned synthetic and real world video pairs with clean background references. The dataset spans diverse photometric, geometric, volumetric, and dynamic interactions, and supports both mask based and instruction driven editing. We further propose CORE, a structured vision language model based evaluation protocol that jointly measures object disappearance and after effect consistency, aligning more closely with human judgments than existing metrics. Benchmarking state of the art methods reveals systematic failures in removing secondary physical effects despite high masked region fidelity, exposing a gap between visual plausibility and causal correctness. BeyondMasks reframes video object removal as causal scene consistency rather than local reconstruction and provides a unified framework for its evaluation.