# Signatures Meet Dynamic Programming: Generalizing Bellman Equations for Trajectory Following

arXiv: https://arxiv.org/abs/2312.05547 · HTML: https://mlsanity.com/paper/2312.05547/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 63 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 19 of 19)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Integrated Task and Motion Planning](https://mlsanity.com/paper/2010.01083/) (2020)
- [A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles](https://mlsanity.com/paper/1604.07446/) (2016)
- [Bilinear Classes: A Structural Framework for Provable Generalization in RL](https://mlsanity.com/paper/2103.10897/) (2021)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Neural Rough Differential Equations for Long Time Series](https://mlsanity.com/paper/2009.08295/) (2020)
- [A Primer on the Signature Method in Machine Learning](https://mlsanity.com/paper/1603.03788/) (2016)
- [Accelerated Policy Learning with Parallel Differentiable Simulation](https://mlsanity.com/paper/2204.07137/) (2022)
- [Kernels for sequentially ordered data](https://mlsanity.com/paper/1601.08169/) (2016)
- [Deep Signature Transforms](https://mlsanity.com/paper/1905.08494/) (2019)
- [The Signature Kernel is the solution of a Goursat PDE](https://mlsanity.com/paper/2006.14794/) (2020)
- [STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulation](https://mlsanity.com/paper/2104.13542/) (2021)
- [Encoding Physical Constraints in Differentiable Newton-Euler Algorithm](https://mlsanity.com/paper/2001.08861/) (2020)
- [Developing the Path Signature Methodology and its Application to Landmark-based Human Action Recognition](https://mlsanity.com/paper/1707.03993/) (2017)
- [Learning Spatial-Semantic Context with Fully Convolutional Recurrent Network for Online Handwritten Chinese Text Recognition](https://mlsanity.com/paper/1610.02616/) (2016)
- [Koopman Spectrum Nonlinear Regulators and Efficient Online Learning](https://mlsanity.com/paper/2106.15775/) (2021)

## Abstract (arXiv)

Path signatures have been proposed as a powerful representation of paths that efficiently captures the path's analytic and geometric characteristics, having useful algebraic properties including fast concatenation of paths through tensor products. Signatures have recently been widely adopted in machine learning problems for time series analysis. In this work we establish connections between value functions typically used in optimal control and intriguing properties of path signatures. These connections motivate our novel control framework with signature transforms that efficiently generalizes the Bellman equation to the space of trajectories. We analyze the properties and advantages of the framework, termed signature control. In particular, we demonstrate that (i) it can naturally deal with varying/adaptive time steps; (ii) it propagates higher-level information more efficiently than value function updates; (iii) it is robust to dynamical system misspecification over long rollouts. As a specific case of our framework, we devise a model predictive control method for path tracking. This method generalizes integral control, being suitable for problems with unknown disturbances. The proposed algorithms are tested in simulation, with differentiable physics models including typical control and robotics tasks such as point-mass, curve following for an ant model, and a robotic manipulator.