# Differentiable Factor Graph Optimization for Learning Smoothers

arXiv: https://arxiv.org/abs/2105.08257 · HTML: https://mlsanity.com/paper/2105.08257/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2021.

## Built on by (most cited first, 17 of 17)

- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [Estimating Body and Hand Motion in an Ego-sensed World](https://mlsanity.com/paper/2410.03665/) (2024)
- [Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstruction](https://mlsanity.com/paper/2409.18121/) (2024)
- [Category-Independent Articulated Object Tracking with Factor Graphs](https://mlsanity.com/paper/2205.03721/) (2022)
- [Confronting Ambiguity in 6D Object Pose Estimation via Score-Based Diffusion on SE(3)](https://mlsanity.com/paper/2305.15873/) (2023)
- [Differentiable Particle Filters through Conditional Normalizing Flow](https://mlsanity.com/paper/2107.00488/) (2021)
- [Revisiting Implicit Differentiation for Learning Problems in Optimal Control](https://mlsanity.com/paper/2310.14468/) (2023)
- [DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning](https://mlsanity.com/paper/2402.05421/) (2024)
- [RL-PGO: Reinforcement Learning-based Planar Pose-Graph Optimization](https://mlsanity.com/paper/2202.13221/) (2022)
- [Streaming Motion Forecasting for Autonomous Driving](https://mlsanity.com/paper/2310.01351/) (2023)
- [DiffPF: Differentiable Particle Filtering with Generative Sampling via Conditional Diffusion Models](https://mlsanity.com/paper/2507.15716/) (2025)
- [Structural Optimization of Factor Graphs for Symbol Detection via Continuous Clustering and Machine Learning](https://mlsanity.com/paper/2211.11406/) (2022)
- [Differentiable SLAM Helps Deep Learning-based LiDAR Perception Tasks](https://mlsanity.com/paper/2309.09206/) (2023)
- [DnD Filter: Differentiable State Estimation for Dynamic Systems using Diffusion Models](https://mlsanity.com/paper/2503.01274/) (2025)
- [Policies over Poses: Reinforcement Learning based Distributed Pose-Graph Optimization for Multi-Robot SLAM](https://mlsanity.com/paper/2510.22740/) (2025)
- [CredibleDFGO: Differentiable Factor Graph Optimization with Credibility Supervision](https://mlsanity.com/paper/2605.06100/) (2026)
- [PixVOD: Pixel-Distributed Direct Visual Odometry and Depth Estimation](https://mlsanity.com/paper/2606.03989/) (2026)

## Abstract (arXiv)

A recent line of work has shown that end-to-end optimization of Bayesian filters can be used to learn state estimators for systems whose underlying models are difficult to hand-design or tune, while retaining the core advantages of probabilistic state estimation. As an alternative approach for state estimation in these settings, we present an end-to-end approach for learning state estimators modeled as factor graph-based smoothers. By unrolling the optimizer we use for maximum a posteriori inference in these probabilistic graphical models, we can learn probabilistic system models in the full context of an overall state estimator, while also taking advantage of the distinct accuracy and runtime advantages that smoothers offer over recursive filters. We study this approach using two fundamental state estimation problems, object tracking and visual odometry, where we demonstrate a significant improvement over existing baselines. Our work comes with an extensive code release, which includes training and evaluation scripts, as well as Python libraries for Lie theory and factor graph optimization: https://sites.google.com/view/diffsmoothing/