# Layered Neural Rendering for Retiming People in Video

arXiv: https://arxiv.org/abs/2009.07833 · HTML: https://mlsanity.com/paper/2009.07833/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2020.

## Built on by (most cited first, 30 of 51)

- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)
- [Text2LIVE: Text-Driven Layered Image and Video Editing](https://mlsanity.com/paper/2204.02491/) (2022)
- [DynIBaR: Neural Dynamic Image-Based Rendering](https://mlsanity.com/paper/2211.11082/) (2022)
- [StableVideo: Text-driven Consistency-aware Diffusion Video Editing](https://mlsanity.com/paper/2308.09592/) (2023)
- [Editable Free-viewpoint Video Using a Layered Neural Representation](https://mlsanity.com/paper/2104.14786/) (2021)
- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [CoDeF: Content Deformation Fields for Temporally Consistent Video Processing](https://mlsanity.com/paper/2308.07926/) (2023)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Deformable Sprites for Unsupervised Video Decomposition](https://mlsanity.com/paper/2204.07151/) (2022)
- [Consistent Depth of Moving Objects in Video](https://mlsanity.com/paper/2108.01166/) (2021)
- [Shape-aware Text-driven Layered Video Editing](https://mlsanity.com/paper/2301.13173/) (2023)
- [Neural Point Light Fields](https://mlsanity.com/paper/2112.01473/) (2021)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [The Emergence of Objectness: Learning Zero-Shot Segmentation from Videos](https://mlsanity.com/paper/2111.06394/) (2021)
- [MULAN: A Multi Layer Annotated Dataset for Controllable Text-to-Image Generation](https://mlsanity.com/paper/2404.02790/) (2024)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [A Survey on Generative AI and LLM for Video Generation, Understanding, and Streaming](https://mlsanity.com/paper/2404.16038/) (2024)
- [MarioNette: Self-Supervised Sprite Learning](https://mlsanity.com/paper/2104.14553/) (2021)
- [Collage Diffusion](https://mlsanity.com/paper/2303.00262/) (2023)
- [Neural Congealing: Aligning Images to a Joint Semantic Atlas](https://mlsanity.com/paper/2302.03956/) (2023)
- [Splatter a Video: Video Gaussian Representation for Versatile Processing](https://mlsanity.com/paper/2406.13870/) (2024)
- [Dynamic Neural Garments](https://mlsanity.com/paper/2102.11811/) (2021)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [Hybrid Neural Fusion for Full-frame Video Stabilization](https://mlsanity.com/paper/2102.06205/) (2021)
- [PAPR: Proximity Attention Point Rendering](https://mlsanity.com/paper/2307.11086/) (2023)
- [OmnimatteZero: Fast Training-free Omnimatte with Pre-trained Video Diffusion Models](https://mlsanity.com/paper/2503.18033/) (2025)
- [Hierarchical Motion Understanding via Motion Programs](https://mlsanity.com/paper/2104.11216/) (2021)
- [OmnimatteRF: Robust Omnimatte with 3D Background Modeling](https://mlsanity.com/paper/2309.07749/) (2023)
- [NeuVV: Neural Volumetric Videos with Immersive Rendering and Editing](https://mlsanity.com/paper/2202.06088/) (2022)

## Abstract (arXiv)

We present a method for retiming people in an ordinary, natural video -- manipulating and editing the time in which different motions of individuals in the video occur. We can temporally align different motions, change the speed of certain actions (speeding up/slowing down, or entirely "freezing" people), or "erase" selected people from the video altogether. We achieve these effects computationally via a dedicated learning-based layered video representation, where each frame in the video is decomposed into separate RGBA layers, representing the appearance of different people in the video. A key property of our model is that it not only disentangles the direct motions of each person in the input video, but also correlates each person automatically with the scene changes they generate -- e.g., shadows, reflections, and motion of loose clothing. The layers can be individually retimed and recombined into a new video, allowing us to achieve realistic, high-quality renderings of retiming effects for real-world videos depicting complex actions and involving multiple individuals, including dancing, trampoline jumping, or group running.