# Stochastic Online Optimization for Cyber-Physical and Robotic Systems

arXiv: https://arxiv.org/abs/2404.05318 · HTML: https://mlsanity.com/paper/2404.05318/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 84 references that are in the map.
- First posted on arXiv 8 Apr 2024.

## Builds on (most cited first, 23 of 23)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Optimizing Neural Networks with Kronecker-factored Approximate Curvature](https://mlsanity.com/paper/1503.05671/) (2015)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Minimizing Finite Sums with the Stochastic Average Gradient](https://mlsanity.com/paper/1309.2388/) (2013)
- [Explore no more: Improved high-probability regret bounds for non-stochastic bandits](https://mlsanity.com/paper/1506.03271/) (2015)
- [Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence](https://mlsanity.com/paper/1505.02250/) (2015)
- [Finite Sample Analysis of Stochastic System Identification](https://mlsanity.com/paper/1903.09122/) (2019)
- [Learning to Play Table Tennis From Scratch using Muscular Robots](https://mlsanity.com/paper/2006.05935/) (2020)
- [RES: Regularized Stochastic BFGS Algorithm](https://mlsanity.com/paper/1401.7625/) (2014)
- [Primal-dual subgradient methods for minimizing uniformly convex functions](https://mlsanity.com/paper/1401.1792/) (2014)
- [Online Primal-Dual Methods with Measurement Feedback for Time-Varying Convex Optimization](https://mlsanity.com/paper/1804.05159/) (2018)
- [Online Learning under Adversarial Nonlinear Constraints](https://mlsanity.com/paper/2306.03655/) (2023)
- [Online Optimization as a Feedback Controller: Stability and Tracking](https://mlsanity.com/paper/1805.09877/) (2018)
- [Iterative Learning Control for Fast and Accurate Position Tracking with an Articulated Soft Robotic Arm](https://mlsanity.com/paper/1901.10187/) (2019)
- [A Fast and Reliable Pick-and-Place Application with a Spherical Soft Robotic Arm](https://mlsanity.com/paper/2011.04624/) (2020)
- [Adaptive Decision-Making with Constraints and Dependent Losses: Performance Guarantees and Applications to Online and Nonlinear Identification](https://mlsanity.com/paper/2304.03321/) (2023)
- [Data-Efficient Online Learning of Ball Placement in Robot Table Tennis](https://mlsanity.com/paper/2308.14562/) (2023)

## Built on by (most cited first, 6 of 6)

- [The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective](https://mlsanity.com/paper/2501.15910/) (2025)
- [Zeroth-Order Optimization Finds Flat Minima](https://mlsanity.com/paper/2506.05454/) (2025)
- [Efficient Model-Based Reinforcement Learning for Robot Control via Online Optimization](https://mlsanity.com/paper/2510.18518/) (2025)
- [Sample-Efficient Model-Free Policy Gradient Methods for Stochastic LQR via Robust Linear Regression](https://mlsanity.com/paper/2512.03764/) (2025)
- [A Stochastic Gradient Descent Approach to Design Policy Gradient Methods for LQR](https://mlsanity.com/paper/2602.18933/) (2026)
- [Shaping Wind-Tunnel Airflow for Unmanned Aerial Vehicles using Online Learning](https://mlsanity.com/paper/2608.03378/) (2026)

## Abstract (arXiv)

We propose a novel gradient-based online optimization framework for solving stochastic programming problems that frequently arise in the context of cyber-physical and robotic systems. Our problem formulation accommodates constraints that model the evolution of a cyber-physical system, which has, in general, a continuous state and action space, is nonlinear, and where the state is only partially observed. We also incorporate an approximate model of the dynamics as prior knowledge into the learning process and show that even rough estimates of the dynamics can significantly improve the convergence of our algorithms. Our online optimization framework encompasses both gradient descent and quasi-Newton methods, and we provide a unified convergence analysis of our algorithms in a non-convex setting. We also characterize the impact of modeling errors in the system dynamics on the convergence rate of the algorithms. Finally, we evaluate our algorithms in simulations of a flexible beam, a four-legged walking robot, and in real-world experiments with a ping-pong playing robot.