# Learning to be Smooth: An End-to-End Differentiable Particle Smoother

arXiv: https://arxiv.org/abs/2502.10546 · HTML: https://mlsanity.com/paper/2502.10546/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 58 references that are in the map.
- First posted on arXiv 14 Feb 2025.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [VIGOR: Cross-View Image Geo-localization beyond One-to-one Retrieval](https://mlsanity.com/paper/2011.12172/) (2020)
- [Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching](https://mlsanity.com/paper/2005.03860/) (2020)
- [Neural Variational Inference and Learning in Belief Networks](https://mlsanity.com/paper/1402.0030/) (2014)
- [Optimal Feature Transport for Cross-View Image Geo-Localization](https://mlsanity.com/paper/1907.05021/) (2019)
- [Beyond Cross-view Image Retrieval: Highly Accurate Vehicle Localization Using Satellite Image](https://mlsanity.com/paper/2204.04752/) (2022)
- [OrienterNet: Visual Localization in 2D Public Maps with Neural Matching](https://mlsanity.com/paper/2304.02009/) (2023)
- [Particle Filter Networks with Application to Visual Localization](https://mlsanity.com/paper/1805.08975/) (2018)
- [Differentiable Particle Filters: End-to-End Learning with Algorithmic Priors](https://mlsanity.com/paper/1805.11122/) (2018)
- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [Visual Cross-View Metric Localization with Dense Uncertainty Estimates](https://mlsanity.com/paper/2208.08519/) (2022)
- [On Particle Methods for Parameter Estimation in State-Space Models](https://mlsanity.com/paper/1412.8695/) (2014)
- [You Are Here: Geolocation by Embedding Maps and Images](https://mlsanity.com/paper/1911.08797/) (2019)
- [Differentiable Particle Filtering without Modifying the Forward Pass](https://mlsanity.com/paper/2106.10314/) (2021)
- [Stochastic Simulation Algorithms for Dynamic Probabilistic Networks](https://mlsanity.com/paper/1302.4965/) (2013)
- [Differentiable and Stable Long-Range Tracking of Multiple Posterior Modes](https://mlsanity.com/paper/2404.08789/) (2024)

## Built on by (most cited first, 4 of 4)

- [PyDPF: A Python Package for Differentiable Particle Filtering](https://mlsanity.com/paper/2510.25693/) (2025)
- [Belief-Conditioned One-Step Diffusion: Real-Time Trajectory Planning with Just-Enough Sensing](https://mlsanity.com/paper/2508.12166/) (2025)
- [BEV-Patch-PF: Particle Filtering with BEV-Aerial Feature Matching for Off-Road Geo-Localization](https://mlsanity.com/paper/2512.15111/) (2025)
- [Efficient Learning of Deep State Space Models via Importance Smoothing](https://mlsanity.com/paper/2605.21108/) (2026)

## Abstract (arXiv)

For challenging state estimation problems arising in domains like vision and robotics, particle-based representations attractively enable temporal reasoning about multiple posterior modes. Particle smoothers offer the potential for more accurate offline data analysis by propagating information both forward and backward in time, but have classically required human-engineered dynamics and observation models. Extending recent advances in discriminative training of particle filters, we develop a framework for low-variance propagation of gradients across long time sequences when training particle smoothers. Our "two-filter'' smoother integrates particle streams that are propagated forward and backward in time, while incorporating stratification and importance weights in the resampling step to provide low-variance gradient estimates for neural network dynamics and observation models. The resulting mixture density particle smoother is substantially more accurate than state-of-the-art particle filters, as well as search-based baselines, for city-scale global vehicle localization from real-world videos and maps.