# 4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion

arXiv: https://arxiv.org/abs/2308.06573 · HTML: https://mlsanity.com/paper/2308.06573/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 49 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [Deep Patch Visual Odometry](https://mlsanity.com/paper/2208.04726/) (2022)
- [FAST-LIO2: Fast Direct LiDAR-inertial Odometry](https://mlsanity.com/paper/2107.06829/) (2021)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [LO-Net: Deep Real-time Lidar Odometry](https://mlsanity.com/paper/1904.08242/) (2019)
- [LVI-SAM: Tightly-coupled Lidar-Visual-Inertial Odometry via Smoothing and Mapping](https://mlsanity.com/paper/2104.10831/) (2021)
- [NeRF-LOAM: Neural Implicit Representation for Large-Scale Incremental LiDAR Odometry and Mapping](https://mlsanity.com/paper/2303.10709/) (2023)
- [Hidden Gems: 4D Radar Scene Flow Learning Using Cross-Modal Supervision](https://mlsanity.com/paper/2303.00462/) (2023)
- [PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimization](https://mlsanity.com/paper/2012.00972/) (2020)
- [Self-Supervised Scene Flow Estimation with 4-D Automotive Radar](https://mlsanity.com/paper/2203.01137/) (2022)
- [Beyond Tracking: Selecting Memory and Refining Poses for Deep Visual Odometry](https://mlsanity.com/paper/1904.01892/) (2019)
- [Efficient 3D Deep LiDAR Odometry](https://mlsanity.com/paper/2111.02135/) (2021)
- [DM-VIO: Delayed Marginalization Visual-Inertial Odometry](https://mlsanity.com/paper/2201.04114/) (2022)
- [LIMO: Lidar-Monocular Visual Odometry](https://mlsanity.com/paper/1807.07524/) (2018)
- [DMLO: Deep Matching LiDAR Odometry](https://mlsanity.com/paper/2004.03796/) (2020)
- [LodoNet: A Deep Neural Network with 2D Keypoint Matchingfor 3D LiDAR Odometry Estimation](https://mlsanity.com/paper/2009.00164/) (2020)
- [DeepPCO: End-to-End Point Cloud Odometry through Deep Parallel Neural Network](https://mlsanity.com/paper/1910.11088/) (2019)
- [4D iRIOM: 4D Imaging Radar Inertial Odometry and Mapping](https://mlsanity.com/paper/2303.13962/) (2023)
- [Self-supervised Visual-LiDAR Odometry with Flip Consistency](https://mlsanity.com/paper/2101.01322/) (2021)
- [Robust Self-Supervised LiDAR Odometry via Representative Structure Discovery and 3D Inherent Error Modeling](https://mlsanity.com/paper/2202.13353/) (2022)
- [LiDAR Data Enrichment Using Deep Learning Based on High-Resolution Image: An Approach to Achieve High-Performance LiDAR SLAM Using Low-cost LiDAR](https://mlsanity.com/paper/2008.03694/) (2020)

## Built on by (most cited first, 11 of 11)

- [RadarOcc: Robust 3D Occupancy Prediction with 4D Imaging Radar](https://mlsanity.com/paper/2405.14014/) (2024)
- [DVLO: Deep Visual-LiDAR Odometry with Local-to-Global Feature Fusion and Bi-Directional Structure Alignment](https://mlsanity.com/paper/2403.18274/) (2024)
- [Doracamom: Joint 3D Detection and Occupancy Prediction with Multi-view 4D Radars and Cameras for Omnidirectional Perception](https://mlsanity.com/paper/2501.15394/) (2025)
- [R2LDM: An Efficient 4D Radar Super-Resolution Framework Leveraging Diffusion Model](https://mlsanity.com/paper/2503.17097/) (2025)
- [4D-ROLLS: 4D Radar Occupancy Learning via LiDAR Supervision](https://mlsanity.com/paper/2505.13905/) (2025)
- [A Survey of Deep Learning Based Radar and Vision Fusion for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/2406.00714/) (2024)
- [DVLO4D: Deep Visual-Lidar Odometry with Sparse Spatial-temporal Fusion](https://mlsanity.com/paper/2509.06023/) (2025)
- [Equi-RO: A 4D mmWave Radar Odometry via Equivariant Networks](https://mlsanity.com/paper/2509.20674/) (2025)
- [DNOI-4DRO: Deep 4D Radar Odometry with Differentiable Neural-Optimization Iterations](https://mlsanity.com/paper/2505.12310/) (2025)
- [Uncertainty-Driven Radar-Inertial Fusion for Instantaneous 3D Ego-Velocity Estimation](https://mlsanity.com/paper/2506.14294/) (2025)
- [MSDNet: Efficient 4D Radar Super-Resolution via Multi-Stage Distillation](https://mlsanity.com/paper/2509.13149/) (2025)

## Abstract (arXiv)

Four-dimensional (4D) radar--visual odometry (4DRVO) integrates complementary information from 4D radar and cameras, making it an attractive solution for achieving accurate and robust pose estimation. However, 4DRVO may exhibit significant tracking errors owing to three main factors: 1) sparsity of 4D radar point clouds; 2) inaccurate data association and insufficient feature interaction between the 4D radar and camera; and 3) disturbances caused by dynamic objects in the environment, affecting odometry estimation. In this paper, we present 4DRVO-Net, which is a method for 4D radar--visual odometry. This method leverages the feature pyramid, pose warping, and cost volume (PWC) network architecture to progressively estimate and refine poses. Specifically, we propose a multi-scale feature extraction network called Radar-PointNet++ that fully considers rich 4D radar point information, enabling fine-grained learning for sparse 4D radar point clouds. To effectively integrate the two modalities, we design an adaptive 4D radar--camera fusion module (A-RCFM) that automatically selects image features based on 4D radar point features, facilitating multi-scale cross-modal feature interaction and adaptive multi-modal feature fusion. In addition, we introduce a velocity-guided point-confidence estimation module to measure local motion patterns, reduce the influence of dynamic objects and outliers, and provide continuous updates during pose refinement. We demonstrate the excellent performance of our method and the effectiveness of each module design on both the VoD and in-house datasets. Our method outperforms all learning-based and geometry-based methods for most sequences in the VoD dataset. Furthermore, it has exhibited promising performance that closely approaches that of the 64-line LiDAR odometry results of A-LOAM without mapping optimization.