# EffiScene: Efficient Per-Pixel Rigidity Inference for Unsupervised Joint Learning of Optical Flow, Depth, Camera Pose and Motion Segmentation

arXiv: https://arxiv.org/abs/2011.08332 · HTML: https://mlsanity.com/paper/2011.08332/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Nov 2020.

## Built on by (most cited first, 22 of 22)

- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [PVO: Panoptic Visual Odometry](https://mlsanity.com/paper/2207.01610/) (2022)
- [A Survey on Deep Stereo Matching in the Twenties](https://mlsanity.com/paper/2407.07816/) (2024)
- [Zero-Shot Monocular Scene Flow Estimation in the Wild](https://mlsanity.com/paper/2501.10357/) (2025)
- [Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes](https://mlsanity.com/paper/2310.18887/) (2023)
- [Self-distilled Feature Aggregation for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2209.07088/) (2022)
- [3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video](https://mlsanity.com/paper/2112.03045/) (2021)
- [FlowSeek: Optical Flow Made Easier with Depth Foundation Models and Motion Bases](https://mlsanity.com/paper/2509.05297/) (2025)
- [D3FlowSLAM: Self-Supervised Dynamic SLAM with Flow Motion Decomposition and DINO Guidance](https://mlsanity.com/paper/2207.08794/) (2022)
- [THE Benchmark: Transferable Representation Learning for Monocular Height Estimation](https://mlsanity.com/paper/2112.14985/) (2021)
- [Globally Consistent Video Depth and Pose Estimation with Efficient Test-Time Training](https://mlsanity.com/paper/2208.02709/) (2022)
- [EMR-MSF: Self-Supervised Recurrent Monocular Scene Flow Exploiting Ego-Motion Rigidity](https://mlsanity.com/paper/2309.01296/) (2023)
- [RaLiFlow: Scene Flow Estimation with 4D Radar and LiDAR Point Clouds](https://mlsanity.com/paper/2512.10376/) (2025)
- [Self-Supervised 3D Scene Flow Estimation and Motion Prediction using Local Rigidity Prior](https://mlsanity.com/paper/2310.11284/) (2023)
- [FlowDepth: Decoupling Optical Flow for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2403.19294/) (2024)
- [AlignDiff: Learning Physically-Grounded Camera Alignment via Diffusion](https://mlsanity.com/paper/2503.21581/) (2025)
- [UFD-PRiME: Unsupervised Joint Learning of Optical Flow and Stereo Depth through Pixel-Level Rigid Motion Estimation](https://mlsanity.com/paper/2310.04712/) (2023)
- [Human Insights Driven Latent Space for Different Driving Perspectives: A Unified Encoder for Efficient Multi-Task Inference](https://mlsanity.com/paper/2409.10095/) (2024)
- [Triangular Consistency as a Universal Constraint for Learning Optical Flow](https://mlsanity.com/paper/2606.19938/) (2026)
- [FlowIt: Global Matching via Hierarchical Transformers and Optimal Transport for Optical Flow](https://mlsanity.com/paper/2603.28759/) (2026)
- [DiffSAC: Diffusion-guided Sampling for Consensus-based Robust Estimation](https://mlsanity.com/paper/2608.30603/) (2026)
- [DynEoMT: Learning Object Dynamicity from Online Segmentation Queries](https://mlsanity.com/paper/2609.14466/) (2026)

## Abstract (arXiv)

This paper addresses the challenging unsupervised scene flow estimation problem by jointly learning four low-level vision sub-tasks: optical flow F, stereo-depth D, camera pose P and motion segmentation S. Our key insight is that the rigidity of the scene shares the same inherent geometrical structure with object movements and scene depth. Hence, rigidity from S can be inferred by jointly coupling F, D and P to achieve more robust estimation. To this end, we propose a novel scene flow framework named EffiScene with efficient joint rigidity learning, going beyond the existing pipeline with independent auxiliary structures. In EffiScene, we first estimate optical flow and depth at the coarse level and then compute camera pose by Perspective-n-Points method. To jointly learn local rigidity, we design a novel Rigidity From Motion (RfM) layer with three principal components: (i) correlation extraction; (ii) boundary learning; and (iii) outlier exclusion. Final outputs are fused based on the rigid map MR from RfM at finer levels. To efficiently train EffiScene, two new losses Lbnd and Lunc are designed to prevent trivial solutions and to regularize the flow boundary discontinuity. Extensive experiments on scene flow benchmark KITTI show that our method is effective and significantly improves the state-of-the-art approaches for all sub-tasks, i.e. optical flow (5.19 rightarrow 4.20), depth estimation (3.78 rightarrow 3.46), visual odometry (0.012 rightarrow 0.011) and motion segmentation (0.57 rightarrow 0.62).