# Bring Event into RGB and LiDAR: Hierarchical Visual-Motion Fusion for Scene Flow arXiv: https://arxiv.org/abs/2403.07432 ยท HTML: https://mlsanity.com/paper/2403.07432/ ## Key facts - Cited by 15 papers in the map (data through 14 Oct 2026). - Builds on 31 of its 45 references that are in the map. - First posted on arXiv 12 Mar 2024. ## Builds on (most cited first, 30 of 31) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014) - [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020) - [Event-based Vision: A Survey](https://mlsanity.com/paper/1904.08405/) (2019) - [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017) - [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016) - [Kindling the Darkness: A Practical Low-light Image Enhancer](https://mlsanity.com/paper/1905.04161/) (2019) - [DSEC: A Stereo Event Camera Dataset for Driving Scenarios](https://mlsanity.com/paper/2103.06011/) (2021) - [FlowFormer: A Transformer Architecture for Optical Flow](https://mlsanity.com/paper/2203.16194/) (2022) - [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021) - [Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation](https://mlsanity.com/paper/1805.09806/) (2018) - [Learning Discrete Representations via Information Maximizing Self-Augmented Training](https://mlsanity.com/paper/1702.08720/) (2017) - [DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency](https://mlsanity.com/paper/1809.01649/) (2018) - [RAFT-3D: Scene Flow using Rigid-Motion Embeddings](https://mlsanity.com/paper/2012.00726/) (2020) - [Event-Based Fusion for Motion Deblurring with Cross-modal Attention](https://mlsanity.com/paper/2112.00167/) (2021) - [Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness](https://mlsanity.com/paper/1608.05842/) (2016) - [Attention-Based Multimodal Fusion for Video Description](https://mlsanity.com/paper/1701.03126/) (2017) - [PV-RAFT: Point-Voxel Correlation Fields for Scene Flow Estimation of Point Clouds](https://mlsanity.com/paper/2012.00987/) (2020) - [SMURF: Self-Teaching Multi-Frame Unsupervised RAFT with Full-Image Warping](https://mlsanity.com/paper/2105.07014/) (2021) - [CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2111.10502/) (2021) - [Deep Rigid Instance Scene Flow](https://mlsanity.com/paper/1904.08913/) (2019) - [RGB-D Saliency Detection via Cascaded Mutual Information Minimization](https://mlsanity.com/paper/2109.07246/) (2021) - [Hidden Gems: 4D Radar Scene Flow Learning Using Cross-Modal Supervision](https://mlsanity.com/paper/2303.00462/) (2023) - [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022) - [Self-Supervised Multi-Frame Monocular Scene Flow](https://mlsanity.com/paper/2105.02216/) (2021) - [SCOOP: Self-Supervised Correspondence and Optimization-Based Scene Flow](https://mlsanity.com/paper/2211.14020/) (2022) - [Mono-SF: Multi-View Geometry Meets Single-View Depth for Monocular Scene Flow Estimation of Dynamic Traffic Scenes](https://mlsanity.com/paper/1908.06316/) (2019) - [RPEFlow: Multimodal Fusion of RGB-PointCloud-Event for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2309.15082/) (2023) - [DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR](https://mlsanity.com/paper/2008.08136/) (2020) - [Self-Supervised 3D Scene Flow Estimation Guided by Superpoints](https://mlsanity.com/paper/2305.02528/) (2023) ## Built on by (most cited first, 15 of 15) - [Recent Event Camera Innovations: A Survey](https://mlsanity.com/paper/2408.13627/) (2024) - [DuCos: Duality Constrained Depth Super-Resolution via Foundation Model](https://mlsanity.com/paper/2503.04171/) (2025) - [Ev-3DOD: Pushing the Temporal Boundaries of 3D Object Detection with Event Cameras](https://mlsanity.com/paper/2502.19630/) (2025) - [Event Camera Meets Mobile Embodied Perception: Abstraction, Algorithm, Acceleration, Application](https://mlsanity.com/paper/2503.22943/) (2025) - [PEOD: A Pixel-Aligned Event-RGB Benchmark for Object Detection under Challenging Conditions](https://mlsanity.com/paper/2511.08140/) (2025) - [Injecting Frame-Event Complementary Fusion into Diffusion for Optical Flow in Challenging Scenes](https://mlsanity.com/paper/2510.10577/) (2025) - [Steering Prediction via a Multi-Sensor System for Autonomous Racing](https://mlsanity.com/paper/2409.19356/) (2024) - [Event-based Sensor Fusion and Application on Odometry: A Survey](https://mlsanity.com/paper/2410.15480/) (2024) - [EMoTive: Event-guided Trajectory Modeling for 3D Motion Estimation](https://mlsanity.com/paper/2503.11371/) (2025) - [Helios 2.0: A Robust, Ultra-Low Power Gesture Recognition System Optimised for Event-Sensor based Wearables](https://mlsanity.com/paper/2503.07825/) (2025) - [x2-Fusion: Cross-Modality and Cross-Dimension Flow Estimation in Event Edge Space](https://mlsanity.com/paper/2603.16671/) (2026) - [High-Quality and Efficient Turbulence Mitigation with Events](https://mlsanity.com/paper/2603.20708/) (2026) - [Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations](https://mlsanity.com/paper/2603.29414/) (2026) - [Weakly Supervised Cross-Modal Learning for 4D Radar Scene Flow Estimation](https://mlsanity.com/paper/2605.18507/) (2026) - [FEMOT: Multi-Object Tracking using Frame and Event Cameras](https://mlsanity.com/paper/2606.14094/) (2026) ## Abstract (arXiv) Single RGB or LiDAR is the mainstream sensor for the challenging scene flow, which relies heavily on visual features to match motion features. Compared with single modality, existing methods adopt a fusion strategy to directly fuse the cross-modal complementary knowledge in motion space. However, these direct fusion methods may suffer the modality gap due to the visual intrinsic heterogeneous nature between RGB and LiDAR, thus deteriorating motion features. We discover that event has the homogeneous nature with RGB and LiDAR in both visual and motion spaces. In this work, we bring the event as a bridge between RGB and LiDAR, and propose a novel hierarchical visual-motion fusion framework for scene flow, which explores a homogeneous space to fuse the cross-modal complementary knowledge for physical interpretation. In visual fusion, we discover that event has a complementarity (relative v.s. absolute) in luminance space with RGB for high dynamic imaging, and has a complementarity (local boundary v.s. global shape) in scene structure space with LiDAR for structure integrity. In motion fusion, we figure out that RGB, event and LiDAR are complementary (spatial-dense, temporal-dense v.s. spatiotemporal-sparse) to each other in correlation space, which motivates us to fuse their motion correlations for motion continuity. The proposed hierarchical fusion can explicitly fuse the multimodal knowledge to progressively improve scene flow from visual space to motion space. Extensive experiments have been performed to verify the superiority of the proposed method.