# Attentive Multimodal Fusion for Optical and Scene Flow

arXiv: https://arxiv.org/abs/2307.15301 · HTML: https://mlsanity.com/paper/2307.15301/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Builds on (most cited first, 25 of 25)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume](https://mlsanity.com/paper/1709.02371/) (2017)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [Learning to Estimate Hidden Motions with Global Motion Aggregation](https://mlsanity.com/paper/2104.02409/) (2021)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [RAFT-3D: Scene Flow using Rigid-Motion Embeddings](https://mlsanity.com/paper/2012.00726/) (2020)
- [MaskFlownet: Asymmetric Feature Matching with Learnable Occlusion Mask](https://mlsanity.com/paper/2003.10955/) (2020)
- [Multispectral Deep Neural Networks for Pedestrian Detection](https://mlsanity.com/paper/1611.02644/) (2016)
- [Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition](https://mlsanity.com/paper/1711.11152/) (2017)
- [CRAFT: Cross-Attentional Flow Transformer for Robust Optical Flow](https://mlsanity.com/paper/2203.16896/) (2022)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation](https://mlsanity.com/paper/2111.10502/) (2021)
- [FlowNet3D++: Geometric Losses For Deep Scene Flow Estimation](https://mlsanity.com/paper/1912.01438/) (2019)
- [On the Integration of Optical Flow and Action Recognition](https://mlsanity.com/paper/1712.08416/) (2017)
- [FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow](https://mlsanity.com/paper/2003.05102/) (2020)
- [LiteFlowNet3: Resolving Correspondence Ambiguity for More Accurate Optical Flow Estimation](https://mlsanity.com/paper/2007.09319/) (2020)
- [DeepLiDARFlow: A Deep Learning Architecture For Scene Flow Estimation Using Monocular Camera and Sparse LiDAR](https://mlsanity.com/paper/2008.08136/) (2020)
- [Flow Fields: Dense Correspondence Fields for Highly Accurate Large Displacement Optical Flow Estimation](https://mlsanity.com/paper/1508.05151/) (2015)
- [Learning Optical Flow and Scene Flow with Bidirectional Camera-LiDAR Fusion](https://mlsanity.com/paper/2303.12017/) (2023)
- [FlowFields++: Accurate Optical Flow Correspondences Meet Robust Interpolation](https://mlsanity.com/paper/1805.03517/) (2018)

## Built on by (most cited first, 3 of 3)

- [Zero-Shot Monocular Scene Flow Estimation in the Wild](https://mlsanity.com/paper/2501.10357/) (2025)
- [MS-RAFT-3D: A Multi-Scale Architecture for Recurrent Image-Based Scene Flow](https://mlsanity.com/paper/2506.01443/) (2025)
- [Multimodal Fusion SLAM with Fourier Attention](https://mlsanity.com/paper/2506.18204/) (2025)

## Abstract (arXiv)

This paper presents an investigation into the estimation of optical and scene flow using RGBD information in scenarios where the RGB modality is affected by noise or captured in dark environments. Existing methods typically rely solely on RGB images or fuse the modalities at later stages, which can result in lower accuracy when the RGB information is unreliable. To address this issue, we propose a novel deep neural network approach named FusionRAFT, which enables early-stage information fusion between sensor modalities (RGB and depth). Our approach incorporates self- and cross-attention layers at different network levels to construct informative features that leverage the strengths of both modalities. Through comparative experiments, we demonstrate that our approach outperforms recent methods in terms of performance on the synthetic dataset Flyingthings3D, as well as the generalization on the real-world dataset KITTI. We illustrate that our approach exhibits improved robustness in the presence of noise and low-lighting conditions that affect the RGB images. We release the code, models and dataset at https://github.com/jiesico/FusionRAFT.