# Particle Filter Networks with Application to Visual Localization

arXiv: https://arxiv.org/abs/1805.08975 · HTML: https://mlsanity.com/paper/1805.08975/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2018.

## Built on by (most cited first, 30 of 65)

- [Differentiable Particle Filtering via Entropy-Regularized Optimal Transport](https://mlsanity.com/paper/2102.07850/) (2021)
- [DiffStack: A Differentiable and Modular Control Stack for Autonomous Vehicles](https://mlsanity.com/paper/2212.06437/) (2022)
- [Differentiable SLAM-net: Learning Particle SLAM for Visual Navigation](https://mlsanity.com/paper/2105.07593/) (2021)
- [INVIGORATE: Interactive Visual Grounding and Grasping in Clutter](https://mlsanity.com/paper/2108.11092/) (2021)
- [Unbiased Asymmetric Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2105.11674/) (2021)
- [LASER: LAtent SpacE Rendering for 2D Visual Localization](https://mlsanity.com/paper/2204.00157/) (2022)
- [An overview of differentiable particle filters for data-adaptive sequential Bayesian inference](https://mlsanity.com/paper/2302.09639/) (2023)
- [The Unreasonable Effectiveness of Pre-Trained Features for Camera Pose Refinement](https://mlsanity.com/paper/2404.10438/) (2024)
- [Differentiable Particle Filtering without Modifying the Forward Pass](https://mlsanity.com/paper/2106.10314/) (2021)
- [F3Loc: Fusion and Filtering for Floorplan Localization](https://mlsanity.com/paper/2403.03370/) (2024)
- [RNN with Particle Flow for Probabilistic Spatio-temporal Forecasting](https://mlsanity.com/paper/2106.06064/) (2021)
- [Differentiable Particle Filters through Conditional Normalizing Flow](https://mlsanity.com/paper/2107.00488/) (2021)
- [BEV-Locator: An End-to-end Visual Semantic Localization Network Using Multi-View Images](https://mlsanity.com/paper/2211.14927/) (2022)
- [Supercharging Floorplan Localization with Semantic Rays](https://mlsanity.com/paper/2507.09291/) (2025)
- [Normalizing Flow-based Differentiable Particle Filters](https://mlsanity.com/paper/2403.01499/) (2024)
- [Efficient Learning of the Parameters of Non-Linear Models using Differentiable Resampling in Particle Filters](https://mlsanity.com/paper/2111.01409/) (2021)
- [Neural Extended Kalman Filters for Learning and Predicting Dynamics of Structural Systems](https://mlsanity.com/paper/2210.04165/) (2022)
- [Unsupervised Learning of Sampling Distributions for Particle Filters](https://mlsanity.com/paper/2302.01174/) (2023)
- [UnLoc: Leveraging Depth Uncertainties for Floorplan Localization](https://mlsanity.com/paper/2509.11301/) (2025)
- [SIDE: State Inference for Partially Observable Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2105.06228/) (2021)
- [Variational Marginal Particle Filters](https://mlsanity.com/paper/2109.15134/) (2021)
- [Conditional Measurement Density Estimation in Sequential Monte Carlo via Normalizing Flow](https://mlsanity.com/paper/2203.08617/) (2022)
- [Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement](https://mlsanity.com/paper/2210.00215/) (2022)
- [SPVLoc: Semantic Panoramic Viewport Matching for 6D Camera Localization in Unseen Environments](https://mlsanity.com/paper/2404.10527/) (2024)
- [LoD-Loc v2: Aerial Visual Localization over Low Level-of-Detail City Models using Explicit Silhouette Alignment](https://mlsanity.com/paper/2507.00659/) (2025)
- [Unsupervised Quality Prediction for Improved Single-Frame and Weighted Sequential Visual Place Recognition](https://mlsanity.com/paper/2307.01464/) (2023)
- [Streaming Motion Forecasting for Autonomous Driving](https://mlsanity.com/paper/2310.01351/) (2023)
- [A Survey of Machine Learning Techniques for Improving Global Navigation Satellite Systems](https://mlsanity.com/paper/2406.16873/) (2024)
- [Enhanced SMC2: Leveraging Gradient Information from Differentiable Particle Filters Within Langevin Proposals](https://mlsanity.com/paper/2407.17296/) (2024)
- [Learning to be Smooth: An End-to-End Differentiable Particle Smoother](https://mlsanity.com/paper/2502.10546/) (2025)

## Abstract (arXiv)

Particle filtering is a powerful approach to sequential state estimation and finds application in many domains, including robot localization, object tracking, etc. To apply particle filtering in practice, a critical challenge is to construct probabilistic system models, especially for systems with complex dynamics or rich sensory inputs such as camera images. This paper introduces the Particle Filter Network (PFnet), which encodes both a system model and a particle filter algorithm in a single neural network. The PF-net is fully differentiable and trained end-to-end from data. Instead of learning a generic system model, it learns a model optimized for the particle filter algorithm. We apply the PF-net to a visual localization task, in which a robot must localize in a rich 3-D world, using only a schematic 2-D floor map. In simulation experiments, PF-net consistently outperforms alternative learning architectures, as well as a traditional model-based method, under a variety of sensor inputs. Further, PF-net generalizes well to new, unseen environments.