# SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.09110 · HTML: https://mlsanity.com/paper/2403.09110/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 45 of its 90 references that are in the map.
- First posted on arXiv 14 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 45)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress](https://mlsanity.com/paper/1806.06877/) (2018)
- [Transfer Learning in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2009.07888/) (2020)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Deep Reinforcement Learning for De-Novo Drug Design](https://mlsanity.com/paper/1711.10907/) (2017)
- [Weak SINDy For Partial Differential Equations](https://mlsanity.com/paper/2007.02848/) (2020)
- [A Tour of Reinforcement Learning: The View from Continuous Control](https://mlsanity.com/paper/1806.09460/) (2018)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Sparse identification of nonlinear dynamics for model predictive control in the low-data limit](https://mlsanity.com/paper/1711.05501/) (2017)
- [Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control](https://mlsanity.com/paper/2111.10992/) (2021)
- [Natural Language Processing: State of The Art, Current Trends and Challenges](https://mlsanity.com/paper/1708.05148/) (2017)
- [Model-Based Reinforcement Learning via Meta-Policy Optimization](https://mlsanity.com/paper/1809.05214/) (2018)
- [Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery](https://mlsanity.com/paper/1912.04825/) (2019)
- [Efficient collective swimming by harnessing vortices through deep reinforcement learning](https://mlsanity.com/paper/1802.02674/) (2018)
- [DeepMoD: Deep learning for Model Discovery in noisy data](https://mlsanity.com/paper/1904.09406/) (2019)

## Built on by (most cited first, 26 of 26)

- [Parametric PDE Control with Deep Reinforcement Learning and Differentiable L0-Sparse Polynomial Policies](https://mlsanity.com/paper/2403.15267/) (2024)
- [Advanced deep-reinforcement-learning methods for flow control: group-invariant and positional-encoding networks improve learning speed and quality](https://mlsanity.com/paper/2407.17822/) (2024)
- [Real-time optimal control of high-dimensional parametrized systems by deep learning-based reduced order models](https://mlsanity.com/paper/2409.05709/) (2024)
- [Explainable AI: Learning from the Learners](https://mlsanity.com/paper/2601.05525/) (2026)
- [HypeRL: Hypernetwork-Based Reinforcement Learning for Control of Parametrized Dynamical Systems](https://mlsanity.com/paper/2501.04538/) (2025)
- [SINDy-KANs: Sparse identification of non-linear dynamics through Kolmogorov-Arnold networks](https://mlsanity.com/paper/2603.18548/) (2026)
- [Interpretable and Efficient Data-driven Discovery and Control of Distributed Systems](https://mlsanity.com/paper/2411.04098/) (2024)
- [Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems](https://mlsanity.com/paper/2504.05588/) (2025)
- [Sparse identification of nonlinear dynamics with library optimization mechanism: Recursive long-term prediction perspective](https://mlsanity.com/paper/2507.18220/) (2025)
- [Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control](https://mlsanity.com/paper/2601.15015/) (2026)
- [Learning from Less: SINDy Surrogates in RL](https://mlsanity.com/paper/2504.18113/) (2025)
- [Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models](https://mlsanity.com/paper/2412.09942/) (2024)
- [SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning](https://mlsanity.com/paper/2502.15512/) (2025)
- [A Dynamical Systems Framework for Reinforcement Learning Safety and Robustness Verification](https://mlsanity.com/paper/2508.15588/) (2025)
- [From STLS to Projection-based Dictionary Selection in Sparse Regression for System Identification](https://mlsanity.com/paper/2512.14404/) (2025)
- [Predicting Dynamics of Ultra-Large Complex Systems by Inferring Governing Equations](https://mlsanity.com/paper/2604.00599/) (2026)
- [Transferable Delay-Aware Reinforcement Learning via Implicit Causal Graph Modeling](https://mlsanity.com/paper/2605.12312/) (2026)
- [How Low Can You Go? Active Learning for Sparse Model Discovery in the Ultra-Low-Data Limit](https://mlsanity.com/paper/2606.12182/) (2026)
- [An Introduction to Sparse Identification of Nonlinear Dynamics for Engineering Applications](https://mlsanity.com/paper/2607.15077/) (2026)
- [Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems](https://mlsanity.com/paper/2607.16177/) (2026)
- [HypEMBER: Hypernetwork-based Ensemble for Robust Policy Learning of Parametrized Dynamical Systems](https://mlsanity.com/paper/2607.19628/) (2026)
- [Gaussian process learning with flow map refinement for parameter estimation in dynamical systems](https://mlsanity.com/paper/2608.22324/) (2026)
- [Learning the Geometry of Admissible Hypotheses through Inductive Bias in Training Distributions](https://mlsanity.com/paper/2608.31028/) (2026)
- [Diagnosing Faults in Reinforcement Learning Simulators and World Models with Canonical Polynomial Invariants](https://mlsanity.com/paper/2609.13194/) (2026)
- [SD-DPC: Sparse Dictionary Differentiable Predictive Control](https://mlsanity.com/paper/2610.02466/) (2026)
- [Arithmetic Actor Heads and Training Stabilization for Out-of-Distribution Reinforcement Learning](https://mlsanity.com/paper/2610.05143/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (DRL) has shown significant promise for uncovering sophisticated control policies that interact in complex environments, such as stabilizing a tokamak fusion reactor or minimizing the drag force on an object in a fluid flow. However, DRL requires an abundance of training examples and may become prohibitively expensive for many applications. In addition, the reliance on deep neural networks often results in an uninterpretable, black-box policy that may be too computationally expensive to use with certain embedded systems. Recent advances in sparse dictionary learning, such as the sparse identification of nonlinear dynamics (SINDy), have shown promise for creating efficient and interpretable data-driven models in the low-data regime. In this work we introduce SINDy-RL, a unifying framework for combining SINDy and DRL to create efficient, interpretable, and trustworthy representations of the dynamics model, reward function, and control policy. We demonstrate the effectiveness of our approaches on benchmark control environments and flow control problems, including gust mitigation on a 3D NACA 0012 airfoil at Re=1000. SINDy-RL achieves comparable performance to modern DRL algorithms using significantly fewer interactions in the environment and results in an interpretable control policy orders of magnitude smaller than a DRL policy.