# Multimodal Sensor Fusion with Differentiable Filters

arXiv: https://arxiv.org/abs/2010.13021 · HTML: https://mlsanity.com/paper/2010.13021/

## Key facts

- Cited by 29 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 25 Oct 2020.

## Built on by (most cited first, 29 of 29)

- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning](https://mlsanity.com/paper/2203.01311/) (2022)
- [Category-Independent Articulated Object Tracking with Factor Graphs](https://mlsanity.com/paper/2205.03721/) (2022)
- [An overview of differentiable particle filters for data-adaptive sequential Bayesian inference](https://mlsanity.com/paper/2302.09639/) (2023)
- [Probabilistic 3D Multi-Object Cooperative Tracking for Autonomous Driving via Differentiable Multi-Sensor Kalman Filter](https://mlsanity.com/paper/2309.14655/) (2023)
- [Learning Vision-based Pursuit-Evasion Robot Policies](https://mlsanity.com/paper/2308.16185/) (2023)
- [RegBN: Batch Normalization of Multimodal Data with Regularization](https://mlsanity.com/paper/2310.00641/) (2023)
- [Learning Sequential Latent Variable Models from Multimodal Time Series Data](https://mlsanity.com/paper/2204.10419/) (2022)
- [Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness](https://mlsanity.com/paper/2205.00001/) (2022)
- [MultiZoo & MultiBench: A Standardized Toolkit for Multimodal Deep Learning](https://mlsanity.com/paper/2306.16413/) (2023)
- [Streaming Motion Forecasting for Autonomous Driving](https://mlsanity.com/paper/2310.01351/) (2023)
- [Multimodal Information Bottleneck for Deep Reinforcement Learning with Multiple Sensors](https://mlsanity.com/paper/2410.17551/) (2024)
- [Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions](https://mlsanity.com/paper/2412.13157/) (2024)
- [Multimodal Learning for Multi-Omics: A Survey](https://mlsanity.com/paper/2211.16509/) (2022)
- [In Defense and Revival of Bayesian Filtering for Thermal Infrared Object Tracking](https://mlsanity.com/paper/2402.17098/) (2024)
- [MultiIoT: Benchmarking Machine Learning for the Internet of Things](https://mlsanity.com/paper/2311.06217/) (2023)
- [Learning active tactile perception through belief-space control](https://mlsanity.com/paper/2312.00215/) (2023)
- [DnD Filter: Differentiable State Estimation for Dynamic Systems using Diffusion Models](https://mlsanity.com/paper/2503.01274/) (2025)
- [The Artificial Neural Twin -- Process Optimization and Continual Learning in Distributed Process Chains](https://mlsanity.com/paper/2403.18343/) (2024)
- [Towards Interpretable Visuo-Tactile Predictive Models for Soft Robot Interactions](https://mlsanity.com/paper/2407.12197/) (2024)
- [Inverse Particle Filter](https://mlsanity.com/paper/2407.16623/) (2024)
- [How Well Do Latent World Models Understand Partially Observable Safety Constraints?](https://mlsanity.com/paper/2510.06492/) (2025)
- [CoCo-InEKF: State Estimation with Learned Contact Covariances in Dynamic, Contact-Rich Scenarios](https://mlsanity.com/paper/2605.15122/) (2026)
- [OVA-IB: One vs All Information Bottleneck for Multi-Modal Alignment](https://mlsanity.com/paper/2605.29900/) (2026)
- [An Attention-based Model for Robust Forecasting with Missing Modality](https://mlsanity.com/paper/2606.13970/) (2026)
- [Physics-Regularized Machine Learning for Proprioceptive Vehicle Localization Using Onboard Sensors](https://mlsanity.com/paper/2607.05663/) (2026)

## Abstract (arXiv)

Leveraging multimodal information with recursive Bayesian filters improves performance and robustness of state estimation, as recursive filters can combine different modalities according to their uncertainties. Prior work has studied how to optimally fuse different sensor modalities with analytical state estimation algorithms. However, deriving the dynamics and measurement models along with their noise profile can be difficult or lead to intractable models. Differentiable filters provide a way to learn these models end-to-end while retaining the algorithmic structure of recursive filters. This can be especially helpful when working with sensor modalities that are high dimensional and have very different characteristics. In contact-rich manipulation, we want to combine visual sensing (which gives us global information) with tactile sensing (which gives us local information). In this paper, we study new differentiable filtering architectures to fuse heterogeneous sensor information. As case studies, we evaluate three tasks: two in planar pushing (simulated and real) and one in manipulating a kinematically constrained door (simulated). In extensive evaluations, we find that differentiable filters that leverage crossmodal sensor information reach comparable accuracies to unstructured LSTM models, while presenting interpretability benefits that may be important for safety-critical systems. We also release an open-source library for creating and training differentiable Bayesian filters in PyTorch, which can be found on our project website: https://sites.google.com/view/multimodalfilter