# Robust Model Predictive Control Design for Autonomous Vehicles with Perception-based Observers

arXiv: https://arxiv.org/abs/2509.05201 · HTML: https://mlsanity.com/paper/2509.05201/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in End-to-end system architecture, one of 9 directions in End-to-end autonomous driving.
- Builds on 12 of its 47 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Placed in

- End-to-end autonomous driving / [End-to-end system architecture](https://mlsanity.com/driving/end-to-end-system-architecture/)

## Builds on (most cited first, 12 of 12)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [End-to-end Autonomous Driving: Challenges and Frontiers](https://mlsanity.com/paper/2306.16927/) (2023)
- [A Survey and Framework of Cooperative Perception: From Heterogeneous Singleton to Hierarchical Cooperation](https://mlsanity.com/paper/2208.10590/) (2022)
- [Robust Guarantees for Perception-Based Control](https://mlsanity.com/paper/1907.03680/) (2019)
- [Certainty Equivalent Perception-Based Control](https://mlsanity.com/paper/2008.12332/) (2020)
- [Set operations and order reductions for constrained zonotopes](https://mlsanity.com/paper/2009.06039/) (2020)
- [Accuracy Prevents Robustness in Perception-based Control](https://mlsanity.com/paper/1910.00119/) (2019)
- [The Implicit Rigid Tube Model Predictive Control](https://mlsanity.com/paper/2306.14543/) (2023)
- [Vision-Driven 2D Supervised Fine-Tuning Framework for Bird's Eye View Perception](https://mlsanity.com/paper/2409.05834/) (2024)
- [RFM-SLAM: Exploiting Relative Feature Measurements to Separate Orientation and Position Estimation in SLAM](https://mlsanity.com/paper/1609.05235/) (2016)
- [A Multi-Scale Isolation Forest Approach for Real-Time Detection and Filtering of FGSM Adversarial Attacks in Video Streams of Autonomous Vehicles](https://mlsanity.com/paper/2502.16044/) (2025)
- [Learning Autonomy: Off-Road Navigation Enhanced by Human Input](https://mlsanity.com/paper/2502.18760/) (2025)

## Abstract (arXiv)

This paper presents a robust model predictive control (MPC) framework that explicitly addresses the non-Gaussian noise inherent in deep learning-based perception modules used for state estimation. Recognizing that accurate uncertainty quantification of the perception module is essential for safe feedback control, our approach departs from the conventional assumption of zero-mean noise quantification of the perception error. Instead, it employs set-based state estimation with constrained zonotopes to capture biased, heavy-tailed uncertainties while maintaining bounded estimation errors. To improve computational efficiency, the robust MPC is reformulated as a linear program (LP), using a Minkowski-Lyapunov-based cost function with an added slack variable to prevent degenerate solutions. Closed-loop stability is ensured through Minkowski-Lyapunov inequalities and contractive zonotopic invariant sets. The largest stabilizing terminal set and its corresponding feedback gain are then derived via an ellipsoidal approximation of the zonotopes. The proposed framework is validated through both simulations and hardware experiments on an omnidirectional mobile robot along with a camera and a convolutional neural network-based perception module implemented within a ROS2 framework. The results demonstrate that the perception-aware MPC provides stable and accurate control performance under heavy-tailed noise conditions, significantly outperforming traditional Gaussian-noise-based designs in terms of both state estimation error bounding and overall control performance.