# The Visual Centrifuge: Model-Free Layered Video Representations

arXiv: https://arxiv.org/abs/1812.01461 · HTML: https://mlsanity.com/paper/1812.01461/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Dec 2018.

## Built on by (most cited first, 23 of 23)

- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers](https://mlsanity.com/paper/2205.13137/) (2022)
- [The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos](https://mlsanity.com/paper/2111.06394/) (2021)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [Bridging Unsupervised and Supervised Depth from Focus via All-in-Focus Supervision](https://mlsanity.com/paper/2108.10843/) (2021)
- [Deep Depth from Focus with Differential Focus Volume](https://mlsanity.com/paper/2112.01712/) (2021)
- [Neural Image Representations for Multi-Image Fusion and Layer Separation](https://mlsanity.com/paper/2108.01199/) (2021)
- [Blind Image Decomposition](https://mlsanity.com/paper/2108.11364/) (2021)
- [Instruction-Guided Visual Masking](https://mlsanity.com/paper/2405.19783/) (2024)
- [A Categorized Reflection Removal Dataset with Diverse Real-world Scenes](https://mlsanity.com/paper/2108.03380/) (2021)
- [Removing Reflections from RAW Photos](https://mlsanity.com/paper/2404.14414/) (2024)
- [Flash-Split: 2D Reflection Removal with Flash Cues and Latent Diffusion Separation](https://mlsanity.com/paper/2501.00637/) (2024)
- [Let's See Clearly: Contaminant Artifact Removal for Moving Cameras](https://mlsanity.com/paper/2104.08852/) (2021)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [SiamTrans: Zero-Shot Multi-Frame Image Restoration with Pre-Trained Siamese Transformers](https://mlsanity.com/paper/2112.09426/) (2021)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Self-supervised Tumor Segmentation through Layer Decomposition](https://mlsanity.com/paper/2109.03230/) (2021)
- [Efficient Flow-Guided Multi-frame De-fencing](https://mlsanity.com/paper/2301.10759/) (2023)
- [Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats](https://mlsanity.com/paper/2410.02764/) (2024)
- [Unsupervised Video Interpolation by Learning Multilayered 2.5D Motion Fields](https://mlsanity.com/paper/2204.09900/) (2022)
- [Principled Reflection Separation via Nonlinear Superposition and Feature Interaction](https://mlsanity.com/paper/2606.02831/) (2026)

## Abstract (arXiv)

True video understanding requires making sense of non-lambertian scenes where the color of light arriving at the camera sensor encodes information about not just the last object it collided with, but about multiple mediums -- colored windows, dirty mirrors, smoke or rain. Layered video representations have the potential of accurately modelling realistic scenes but have so far required stringent assumptions on motion, lighting and shape. Here we propose a learning-based approach for multi-layered video representation: we introduce novel uncertainty-capturing 3D convolutional architectures and train them to separate blended videos. We show that these models then generalize to single videos, where they exhibit interesting abilities: color constancy, factoring out shadows and separating reflections. We present quantitative and qualitative results on real world videos.