# DytanVO: Joint Refinement of Visual Odometry and Motion Segmentation in Dynamic Environments

arXiv: https://arxiv.org/abs/2209.08430 · HTML: https://mlsanity.com/paper/2209.08430/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 17 Sep 2022.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Direct Sparse Odometry](https://mlsanity.com/paper/1607.02565/) (2016)
- [GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose](https://mlsanity.com/paper/1803.02276/) (2018)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks](https://mlsanity.com/paper/1609.05130/) (2016)
- [DynaSLAM: Tracking, Mapping and Inpainting in Dynamic Scenes](https://mlsanity.com/paper/1806.05620/) (2018)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments](https://mlsanity.com/paper/1809.08379/) (2018)
- [DeepTAM: Deep Tracking and Mapping](https://mlsanity.com/paper/1808.01900/) (2018)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [MID-Fusion: Octree-based Object-Level Multi-Instance Dynamic SLAM](https://mlsanity.com/paper/1812.07976/) (2018)
- [Deep Auxiliary Learning for Visual Localization and Odometry](https://mlsanity.com/paper/1803.03642/) (2018)
- [Visual Odometry Revisited: What Should Be Learnt?](https://mlsanity.com/paper/1909.09803/) (2019)
- [Learning to Segment Rigid Motions from Two Frames](https://mlsanity.com/paper/2101.03694/) (2021)
- [VDO-SLAM: A Visual Dynamic Object-aware SLAM System](https://mlsanity.com/paper/2005.11052/) (2020)
- [FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow](https://mlsanity.com/paper/2003.05102/) (2020)
- [Milking CowMask for Semi-Supervised Image Classification](https://mlsanity.com/paper/2003.12022/) (2020)
- [Self-Supervised Deep Visual Odometry with Online Adaptation](https://mlsanity.com/paper/2005.06136/) (2020)
- [AirDOS: Dynamic SLAM benefits from Articulated Objects](https://mlsanity.com/paper/2109.09903/) (2021)
- [Monocular SLAM Supported Object Recognition](https://mlsanity.com/paper/1506.01732/) (2015)

## Built on by (most cited first, 27 of 27)

- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos](https://mlsanity.com/paper/2412.04463/) (2024)
- [Stereo4D: Learning How Things Move in 3D from Internet Stereo Videos](https://mlsanity.com/paper/2412.09621/) (2024)
- [NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild](https://mlsanity.com/paper/2405.18715/) (2024)
- [Deep Patch Visual SLAM](https://mlsanity.com/paper/2408.01654/) (2024)
- [LEAP-VO: Long-term Effective Any Point Tracking for Visual Odometry](https://mlsanity.com/paper/2401.01887/) (2024)
- [Dynamic Camera Poses and Where to Find Them](https://mlsanity.com/paper/2504.17788/) (2025)
- [Back on Track: Bundle Adjustment for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2504.14516/) (2025)
- [DynaMoN: Motion-Aware Fast and Robust Camera Localization for Dynamic Neural Radiance Fields](https://mlsanity.com/paper/2309.08927/) (2023)
- [Reconstructing 4D Spatial Intelligence: A Survey](https://mlsanity.com/paper/2507.21045/) (2025)
- [DynaGSLAM: Real-Time Gaussian-Splatting SLAM for Online Rendering, Tracking, Motion Predictions of Moving Objects in Dynamic Scenes](https://mlsanity.com/paper/2503.11979/) (2025)
- [DVI-SLAM: A Dual Visual Inertial SLAM Network](https://mlsanity.com/paper/2309.13814/) (2023)
- [Scene-agnostic Pose Regression for Visual Localization](https://mlsanity.com/paper/2503.19543/) (2025)
- [MambaVO: Deep Visual Odometry Based on Sequential Matching Refinement and Training Smoothing](https://mlsanity.com/paper/2412.20082/) (2024)
- [Video Understanding: From Geometry and Semantics to Unified Models](https://mlsanity.com/paper/2603.17840/) (2026)
- [Dy3DGS-SLAM: Monocular 3D Gaussian Splatting SLAM for Dynamic Environments](https://mlsanity.com/paper/2506.05965/) (2025)
- [SmallGS: Gaussian Splatting-based Camera Pose Estimation for Small-Baseline Videos](https://mlsanity.com/paper/2504.17810/) (2025)
- [ProDyG: Progressive Dynamic Scene Reconstruction via Gaussian Splatting from Monocular Videos](https://mlsanity.com/paper/2509.17864/) (2025)
- [Dynamic Visual SLAM using a General 3D Prior](https://mlsanity.com/paper/2512.06868/) (2025)
- [PoseFM: Relative Camera Pose Estimation Through Flow Matching](https://mlsanity.com/paper/2604.22350/) (2026)
- [MVOFormer: Flow-Semantic Transformer for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2606.16474/) (2026)
- [Neural Rendering based Urban Scene Reconstruction for Autonomous Driving](https://mlsanity.com/paper/2402.06826/) (2024)
- [Str-L Pose: Integrating Point and Structured Line for Relative Pose Estimation in Dual-Graph](https://mlsanity.com/paper/2408.15750/) (2024)
- [LGU-SLAM: Learnable Gaussian Uncertainty Matching with Deformable Correlation Sampling for Deep Visual SLAM](https://mlsanity.com/paper/2410.23231/) (2024)
- [SPORTS: Simultaneous Panoptic Odometry, Rendering, Tracking and Segmentation for Urban Scenes Understanding](https://mlsanity.com/paper/2510.12749/) (2025)
- [LST-SLAM: A Stereo Thermal SLAM System for Kilometer-Scale Dynamic Environments](https://mlsanity.com/paper/2602.20925/) (2026)
- [RoSe-SLAM: Robust Semantic-Aware Gaussian Splatting SLAM from Dynamic Monocular Videos](https://mlsanity.com/paper/2608.29003/) (2026)

## Abstract (arXiv)

Learning-based visual odometry (VO) algorithms achieve remarkable performance on common static scenes, benefiting from high-capacity models and massive annotated data, but tend to fail in dynamic, populated environments. Semantic segmentation is largely used to discard dynamic associations before estimating camera motions but at the cost of discarding static features and is hard to scale up to unseen categories. In this paper, we leverage the mutual dependence between camera ego-motion and motion segmentation and show that both can be jointly refined in a single learning-based framework. In particular, we present DytanVO, the first supervised learning-based VO method that deals with dynamic environments. It takes two consecutive monocular frames in real-time and predicts camera ego-motion in an iterative fashion. Our method achieves an average improvement of 27.7% in ATE over state-of-the-art VO solutions in real-world dynamic environments, and even performs competitively among dynamic visual SLAM systems which optimize the trajectory on the backend. Experiments on plentiful unseen environments also demonstrate our method's generalizability.