# Controllable Attention for Structured Layered Video Decomposition

arXiv: https://arxiv.org/abs/1910.11306 · HTML: https://mlsanity.com/paper/1910.11306/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2019.

## Built on by (most cited first, 7 of 7)

- [Self-supervised Video Object Segmentation by Motion Grouping](https://mlsanity.com/paper/2104.07658/) (2021)
- [Segmenting Moving Objects via an Object-Centric Layered Representation](https://mlsanity.com/paper/2207.02206/) (2022)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Generative Omnimatte: Learning to Decompose Video into Layers](https://mlsanity.com/paper/2411.16683/) (2024)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Hashing Neural Video Decomposition with Multiplicative Residuals in Space-Time](https://mlsanity.com/paper/2309.14022/) (2023)

## Abstract (arXiv)

The objective of this paper is to be able to separate a video into its natural layers, and to control which of the separated layers to attend to. For example, to be able to separate reflections, transparency or object motion. We make the following three contributions: (i) we introduce a new structured neural network architecture that explicitly incorporates layers (as spatial masks) into its design. This improves separation performance over previous general purpose networks for this task; (ii) we demonstrate that we can augment the architecture to leverage external cues such as audio for controllability and to help disambiguation; and (iii) we experimentally demonstrate the effectiveness of our approach and training procedure with controlled experiments while also showing that the proposed model can be successfully applied to real-word applications such as reflection removal and action recognition in cluttered scenes.