# MPI-Flow: Learning Realistic Optical Flow with Multiplane Images

arXiv: https://arxiv.org/abs/2309.06714 · HTML: https://mlsanity.com/paper/2309.06714/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 52 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://mlsanity.com/paper/2103.14910/) (2021)
- [Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation](https://mlsanity.com/paper/1904.05290/) (2019)
- [What Matters in Unsupervised Optical Flow](https://mlsanity.com/paper/2006.04902/) (2020)
- [Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation](https://mlsanity.com/paper/2003.13045/) (2020)
- [AutoFlow: Learning a Better Training Set for Optical Flow](https://mlsanity.com/paper/2104.14544/) (2021)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)
- [Layer-structured 3D Scene Inference via View Synthesis](https://mlsanity.com/paper/1807.10264/) (2018)
- [PixelSynth: Generating a 3D-Consistent Experience from a Single Image](https://mlsanity.com/paper/2108.05892/) (2021)
- [Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images](https://mlsanity.com/paper/2205.11733/) (2022)
- [UPFlow: Upsampling Pyramid for Unsupervised Optical Flow Learning](https://mlsanity.com/paper/2012.00212/) (2020)
- [FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow](https://mlsanity.com/paper/2003.05102/) (2020)
- [Learning Optical Flow with Adaptive Graph Reasoning](https://mlsanity.com/paper/2202.03857/) (2022)
- [Learning optical flow from still images](https://mlsanity.com/paper/2104.03965/) (2021)
- [RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos](https://mlsanity.com/paper/2207.11075/) (2022)
- [Learning monocular 3D reconstruction of articulated categories from motion](https://mlsanity.com/paper/2103.16352/) (2021)
- [Multiple Object Tracking by Flowing and Fusing](https://mlsanity.com/paper/2001.11180/) (2020)
- [DistractFlow: Improving Optical Flow Estimation via Realistic Distractions and Pseudo-Labeling](https://mlsanity.com/paper/2303.14078/) (2023)
- [Semi-Supervised Learning of Optical Flow by Flow Supervisor](https://mlsanity.com/paper/2207.10314/) (2022)
- [Cross-MPI: Cross-scale Stereo for Image Super-Resolution using Multiplane Images](https://mlsanity.com/paper/2011.14631/) (2020)
- [Object Tracking in Satellite Videos Based on a Multi-Frame Optical Flow Tracker](https://mlsanity.com/paper/1804.09323/) (2018)
- [Learning multiplane images from single views with self-supervision](https://mlsanity.com/paper/2110.09380/) (2021)

## Built on by (most cited first, 7 of 7)

- [Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance](https://mlsanity.com/paper/2505.02109/) (2025)
- [Distilling Monocular Foundation Model for Fine-grained Depth Completion](https://mlsanity.com/paper/2503.16970/) (2025)
- [StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences](https://mlsanity.com/paper/2311.17099/) (2023)
- [Enhancing Unregistered Hyperspectral Image Super-Resolution via Unmixing-based Abundance Fusion Learning](https://mlsanity.com/paper/2603.07918/) (2026)
- [Learning Dense Feature Matching via Lifting Single 2D Image to 3D Space](https://mlsanity.com/paper/2507.00392/) (2025)
- [Boosting Zero-shot Stereo Matching using Large-scale Mixed Images Sources in the Real World](https://mlsanity.com/paper/2505.08607/) (2025)
- [Rethinking Unsupervised Cross-modal Flow Estimation: Learning from Decoupled Optimization and Consistency Constraint](https://mlsanity.com/paper/2509.24423/) (2025)

## Abstract (arXiv)

The accuracy of learning-based optical flow estimation models heavily relies on the realism of the training datasets. Current approaches for generating such datasets either employ synthetic data or generate images with limited realism. However, the domain gap of these data with real-world scenes constrains the generalization of the trained model to real-world applications. To address this issue, we investigate generating realistic optical flow datasets from real-world images. Firstly, to generate highly realistic new images, we construct a layered depth representation, known as multiplane images (MPI), from single-view images. This allows us to generate novel view images that are highly realistic. To generate optical flow maps that correspond accurately to the new image, we calculate the optical flows of each plane using the camera matrix and plane depths. We then project these layered optical flows into the output optical flow map with volume rendering. Secondly, to ensure the realism of motion, we present an independent object motion module that can separate the camera and dynamic object motion in MPI. This module addresses the deficiency in MPI-based single-view methods, where optical flow is generated only by camera motion and does not account for any object movement. We additionally devise a depth-aware inpainting module to merge new images with dynamic objects and address unnatural motion occlusions. We show the superior performance of our method through extensive experiments on real-world datasets. Moreover, our approach achieves state-of-the-art performance in both unsupervised and supervised training of learning-based models. The code will be made publicly available at: urlhttps://github.com/Sharpiless/MPI-Flow.