# OptiState: State Estimation of Legged Robots using Gated Networks with Transformer-based Vision and Kalman Filtering

arXiv: https://arxiv.org/abs/2401.16719 · HTML: https://mlsanity.com/paper/2401.16719/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 25 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Builds on (most cited first, 11 of 11)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics](https://mlsanity.com/paper/2107.10043/) (2021)
- [Combining Generative and Discriminative Models for Hybrid Inference](https://mlsanity.com/paper/1906.02547/) (2019)
- [AI-IMU Dead-Reckoning](https://mlsanity.com/paper/1904.06064/) (2019)
- [Learning Inertial Odometry for Dynamic Legged Robot State Estimation](https://mlsanity.com/paper/2111.00789/) (2021)
- [Cerberus: Low-Drift Visual-Inertial-Leg Odometry For Agile Locomotion](https://mlsanity.com/paper/2209.07654/) (2022)
- [IMU Data Processing For Inertial Aided Navigation: A Recurrent Neural Network Based Approach](https://mlsanity.com/paper/2103.14286/) (2021)
- [Robust Legged Robot State Estimation Using Factor Graph Optimization](https://mlsanity.com/paper/1904.03048/) (2019)
- [Real-to-Sim: Predicting Residual Errors of Robotic Systems with Sparse Data using a Learning-based Unscented Kalman Filter](https://mlsanity.com/paper/2209.03210/) (2022)
- [SABER: Data-Driven Motion Planner for Autonomously Navigating Heterogeneous Robots](https://mlsanity.com/paper/2108.01262/) (2021)
- [SCALER: A Tough Versatile Quadruped Free-Climber Robot](https://mlsanity.com/paper/2207.01180/) (2022)

## Abstract (arXiv)

State estimation for legged robots is challenging due to their highly dynamic motion and limitations imposed by sensor accuracy. By integrating Kalman filtering, optimization, and learning-based modalities, we propose a hybrid solution that combines proprioception and exteroceptive information for estimating the state of the robot's trunk. Leveraging joint encoder and IMU measurements, our Kalman filter is enhanced through a single-rigid body model that incorporates ground reaction force control outputs from convex Model Predictive Control optimization. The estimation is further refined through Gated Recurrent Units, which also considers semantic insights and robot height from a Vision Transformer autoencoder applied on depth images. This framework not only furnishes accurate robot state estimates, including uncertainty evaluations, but can minimize the nonlinear errors that arise from sensor measurements and model simplifications through learning. The proposed methodology is evaluated in hardware using a quadruped robot on various terrains, yielding a 65% improvement on the Root Mean Squared Error compared to our VIO SLAM baseline. Code example: https://github.com/AlexS28/OptiState