# OmnimatteRF: Robust Omnimatte with 3D Background Modeling

arXiv: https://arxiv.org/abs/2309.07749 · HTML: https://mlsanity.com/paper/2309.07749/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 14 Sep 2023.

## Builds on (most cited first, 25 of 25)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [Space-time Neural Irradiance Fields for Free-Viewpoint Video](https://mlsanity.com/paper/2011.12950/) (2020)
- [Robust Dynamic Radiance Fields](https://mlsanity.com/paper/2301.02239/) (2023)
- [HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling](https://mlsanity.com/paper/2301.02238/) (2023)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [D2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video](https://mlsanity.com/paper/2205.15838/) (2022)
- [Deformable Sprites for Unsupervised Video Decomposition](https://mlsanity.com/paper/2204.07151/) (2022)
- [Shape-aware Text-driven Layered Video Editing](https://mlsanity.com/paper/2301.13173/) (2023)
- [Layered Neural Rendering for Retiming People in Video](https://mlsanity.com/paper/2009.07833/) (2020)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Learning Neural Light Fields with Ray-Space Embedding Networks](https://mlsanity.com/paper/2112.01523/) (2021)
- [Instance Shadow Detection](https://mlsanity.com/paper/1911.07034/) (2019)
- [Eikonal Fields for Refractive Novel-View Synthesis](https://mlsanity.com/paper/2202.00948/) (2022)
- [FactorMatte: Redefining Video Matting for Re-Composition Tasks](https://mlsanity.com/paper/2211.02145/) (2022)

## Built on by (most cited first, 14 of 14)

- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2503.18033/) (2025)
- [TransPixeler: Advancing Text-to-Video Generation with Transparency](https://mlsanity.com/paper/2501.03006/) (2025)
- [O-DisCo-Edit: Object Distortion Control for Unified Realistic Video Editing](https://mlsanity.com/paper/2509.01596/) (2025)
- [VOID: Video Object and Interaction Deletion](https://mlsanity.com/paper/2604.02296/) (2026)
- [Object-WIPER : Training-Free Object and Associated Effect Removal in Videos](https://mlsanity.com/paper/2601.06391/) (2026)
- [OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos](https://mlsanity.com/paper/2404.00676/) (2024)
- [Generative Video Matting](https://mlsanity.com/paper/2508.07905/) (2025)
- [Over++: Generative Video Compositing for Layer Interaction Effects](https://mlsanity.com/paper/2512.19661/) (2025)
- [Neural Atlas Graphs for Dynamic Scene Decomposition and Editing](https://mlsanity.com/paper/2509.16336/) (2025)
- [Efficient Portrait Matte Creation With Layer Diffusion and Connectivity Priors](https://mlsanity.com/paper/2501.16147/) (2025)
- [Generating Humanless Environment Walkthroughs from Egocentric Walking Tour Videos](https://mlsanity.com/paper/2603.29036/) (2026)
- [Explicit Layer Modeling for Video Object Insertion and Video Layer Decomposition](https://mlsanity.com/paper/2607.25802/) (2026)
- [VOR-Bench: A Human Perception-Driven Benchmark for Video Object Removal](https://mlsanity.com/paper/2609.16878/) (2026)

## Abstract (arXiv)

Video matting has broad applications, from adding interesting effects to casually captured movies to assisting video production professionals. Matting with associated effects such as shadows and reflections has also attracted increasing research activity, and methods like Omnimatte have been proposed to separate dynamic foreground objects of interest into their own layers. However, prior works represent video backgrounds as 2D image layers, limiting their capacity to express more complicated scenes, thus hindering application to real-world videos. In this paper, we propose a novel video matting method, OmnimatteRF, that combines dynamic 2D foreground layers and a 3D background model. The 2D layers preserve the details of the subjects, while the 3D background robustly reconstructs scenes in real-world videos. Extensive experiments demonstrate that our method reconstructs scenes with better quality on various videos.