# DVLO: Deep Visual-LiDAR Odometry with Local-to-Global Feature Fusion and Bi-Directional Structure Alignment

arXiv: https://arxiv.org/abs/2403.18274 · HTML: https://mlsanity.com/paper/2403.18274/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 62 references that are in the map.
- First posted on arXiv 27 Mar 2024.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM](https://mlsanity.com/paper/2007.11898/) (2020)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [FlowNet3D: Learning Scene Flow in 3D Point Clouds](https://mlsanity.com/paper/1806.01411/) (2018)
- [DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks](https://mlsanity.com/paper/1709.08429/) (2017)
- [EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection](https://mlsanity.com/paper/2007.08856/) (2020)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusion](https://mlsanity.com/paper/2303.03595/) (2023)
- [LO-Net: Deep Real-time Lidar Odometry](https://mlsanity.com/paper/1904.08242/) (2019)
- [CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2111.10502/) (2021)
- [F-LOAM: Fast LiDAR Odometry And Mapping](https://mlsanity.com/paper/2107.00822/) (2021)
- [Image as Set of Points](https://mlsanity.com/paper/2303.01494/) (2023)
- [NeRF-LOAM: Neural Implicit Representation for Large-Scale Incremental LiDAR Odometry and Mapping](https://mlsanity.com/paper/2303.10709/) (2023)
- [RegFormer: An Efficient Projection-Aware Transformer Network for Large-Scale Point Cloud Registration](https://mlsanity.com/paper/2303.12384/) (2023)
- [PWCLO-Net: Deep LiDAR Odometry in 3D Point Clouds Using Hierarchical Embedding Mask Optimization](https://mlsanity.com/paper/2012.00972/) (2020)
- [LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localization](https://mlsanity.com/paper/2312.16648/) (2023)
- [Efficient 3D Deep LiDAR Odometry](https://mlsanity.com/paper/2111.02135/) (2021)
- [SelfVIO: Self-Supervised Deep Monocular Visual-Inertial Odometry and Depth Estimation](https://mlsanity.com/paper/1911.09968/) (2019)
- [DF-VO: What Should Be Learnt for Visual Odometry?](https://mlsanity.com/paper/2103.00933/) (2021)
- [LIMO: Lidar-Monocular Visual Odometry](https://mlsanity.com/paper/1807.07524/) (2018)
- [Occlusion Guided Scene Flow Estimation on 3D Point Clouds](https://mlsanity.com/paper/2011.14880/) (2020)
- [DeepPCO: End-to-End Point Cloud Odometry through Deep Parallel Neural Network](https://mlsanity.com/paper/1910.11088/) (2019)
- [DELFlow: Dense Efficient Learning of Scene Flow for Large-Scale Point Clouds](https://mlsanity.com/paper/2308.04383/) (2023)
- [Approaches, Challenges, and Applications for Deep Visual Odometry: Toward to Complicated and Emerging Areas](https://mlsanity.com/paper/2009.02672/) (2020)

## Built on by (most cited first, 26 of 26)

- [DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenes](https://mlsanity.com/paper/2411.11921/) (2024)
- [ReTrack: Evidence-Driven Dual-Stream Directional Anchor Calibration Network for Composed Video Retrieval](https://mlsanity.com/paper/2604.17898/) (2026)
- [NeSLAM: Neural Implicit Mapping and Self-Supervised Feature Tracking With Depth Completion and Denoising](https://mlsanity.com/paper/2403.20034/) (2024)
- [DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving](https://mlsanity.com/paper/2412.09043/) (2024)
- [INTENT: Invariance and Discrimination-aware Noise Mitigation for Robust Composed Image Retrieval](https://mlsanity.com/paper/2604.18051/) (2026)
- [HABIT: Chrono-Synergia Robust Progressive Learning Framework for Composed Image Retrieval](https://mlsanity.com/paper/2604.18037/) (2026)
- [Talk2Radar: Bridging Natural Language with 4D mmWave Radar for 3D Referring Expression Comprehension](https://mlsanity.com/paper/2405.12821/) (2024)
- [VPGS-SLAM: Voxel-based Progressive 3D Gaussian SLAM in Large-Scale Scenes](https://mlsanity.com/paper/2505.18992/) (2025)
- [TurboReg: TurboClique for Robust and Efficient Point Cloud Registration](https://mlsanity.com/paper/2507.01439/) (2025)
- [A Lesson in Splats: Teacher-Guided Diffusion for 3D Gaussian Splats Generation with 2D Supervision](https://mlsanity.com/paper/2412.00623/) (2024)
- [UniBEVFusion: Unified Radar-Vision BEVFusion for 3D Object Detection](https://mlsanity.com/paper/2409.14751/) (2024)
- [RoGs: Large Scale Road Surface Reconstruction with Meshgrid Gaussian](https://mlsanity.com/paper/2405.14342/) (2024)
- [OG-Gaussian: Occupancy Based Street Gaussians for Autonomous Driving](https://mlsanity.com/paper/2502.14235/) (2025)
- [4DSTR: Advancing Generative 4D Gaussians with Spatial-Temporal Rectification for High-Quality and Consistent 4D Generation](https://mlsanity.com/paper/2511.07241/) (2025)
- [Perception Graph for Cognitive Attack Reasoning in Augmented Reality](https://mlsanity.com/paper/2509.05324/) (2025)
- [NeuroGauss4D-PCI: 4D Neural Fields and Gaussian Deformation Fields for Point Cloud Interpolation](https://mlsanity.com/paper/2405.14241/) (2024)
- [Generalizable Prompt Learning of CLIP: A Brief Overview](https://mlsanity.com/paper/2503.01263/) (2025)
- [DVLO4D: Deep Visual-Lidar Odometry with Sparse Spatial-temporal Fusion](https://mlsanity.com/paper/2509.06023/) (2025)
- [RegFormer++: An Efficient Large-Scale 3D LiDAR Point Registration Network with Projection-Aware 2D Transformer](https://mlsanity.com/paper/2603.14290/) (2026)
- [FAST: A Synergistic Framework of Attention and State-space Models for Spatiotemporal Traffic Prediction](https://mlsanity.com/paper/2604.13453/) (2026)
- [Enhancing Target-unspecific Tasks through a Features Matrix](https://mlsanity.com/paper/2505.03414/) (2025)
- [DNOI-4DRO: Deep 4D Radar Odometry with Differentiable Neural-Optimization Iterations](https://mlsanity.com/paper/2505.12310/) (2025)
- [EADReg: Probabilistic Correspondence Generation with Efficient Autoregressive Diffusion Model for Outdoor Point Cloud Registration](https://mlsanity.com/paper/2411.15271/) (2024)
- [SGLoc: Semantic Localization System for Camera Pose Estimation from 3D Gaussian Splatting Representation](https://mlsanity.com/paper/2507.12027/) (2025)
- [Dense-depth map guided deep Lidar-Visual Odometry with Sparse Point Clouds and Images](https://mlsanity.com/paper/2507.15496/) (2025)
- [MemReg: Streaming Outdoor LiDAR Point Cloud Registration with Hybrid Memory Buffers](https://mlsanity.com/paper/2610.12624/) (2026)

## Abstract (arXiv)

Information inside visual and LiDAR data is well complementary derived from the fine-grained texture of images and massive geometric information in point clouds. However, it remains challenging to explore effective visual-LiDAR fusion, mainly due to the intrinsic data structure inconsistency between two modalities: Image pixels are regular and dense, but LiDAR points are unordered and sparse. To address the problem, we propose a local-to-global fusion network (DVLO) with bi-directional structure alignment. To obtain locally fused features, we project points onto the image plane as cluster centers and cluster image pixels around each center. Image pixels are pre-organized as pseudo points for image-to-point structure alignment. Then, we convert points to pseudo images by cylindrical projection (point-to-image structure alignment) and perform adaptive global feature fusion between point features and local fused features. Our method achieves state-of-the-art performance on KITTI odometry and FlyingThings3D scene flow datasets compared to both single-modal and multi-modal methods. Codes are released at https://github.com/IRMVLab/DVLO.