# Learning Covariances for Estimation with Constrained Bilevel Optimization

arXiv: https://arxiv.org/abs/2309.09718 · HTML: https://mlsanity.com/paper/2309.09718/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Sep 2023.

## Built on by (most cited first, 4 of 4)

- [Your Learned Constraint is Secretly a Backward Reachable Tube](https://mlsanity.com/paper/2501.15618/) (2025)
- [Acoustic Neural 3D Reconstruction Under Pose Drift](https://mlsanity.com/paper/2503.08930/) (2025)
- [TouchAnything: Diffusion-Guided 3D Reconstruction from Sparse Robot Touches](https://mlsanity.com/paper/2604.08945/) (2026)
- [Adapting Vision Foundation Models to Acoustics for Pose-Free 3D Sonar Reconstruction](https://mlsanity.com/paper/2609.06261/) (2026)

## Abstract (arXiv)

We consider the problem of learning error covariance matrices for robotic state estimation. The convergence of a state estimator to the correct belief over the robot state is dependent on the proper tuning of noise models. During inference, these models are used to weigh different blocks of the Jacobian and error vector resulting from linearization and hence, additionally affect the stability and convergence of the non-linear system. We propose a gradient-based method to estimate well-conditioned covariance matrices by formulating the learning process as a constrained bilevel optimization problem over factor graphs. We evaluate our method against baselines across a range of simulated and real-world tasks and demonstrate that our technique converges to model estimates that lead to better solutions as evidenced by the improved tracking accuracy on unseen test trajectories.