# ODE-based Recurrent Model-free Reinforcement Learning for POMDPs

arXiv: https://arxiv.org/abs/2309.14078 · HTML: https://mlsanity.com/paper/2309.14078/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 52 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 30 of 36)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Characterizing possible failure modes in physics-informed neural networks](https://mlsanity.com/paper/2109.01050/) (2021)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Neural Controlled Differential Equations for Irregular Time Series](https://mlsanity.com/paper/2005.08926/) (2020)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [Human-level performance in first-person multiplayer games with population-based deep reinforcement learning](https://mlsanity.com/paper/1807.01281/) (2018)
- [GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series](https://mlsanity.com/paper/1905.12374/) (2019)
- [Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model](https://mlsanity.com/paper/1907.00953/) (2019)
- [VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning](https://mlsanity.com/paper/1910.08348/) (2019)
- [Latent ODEs for Irregularly-Sampled Time Series](https://mlsanity.com/paper/1907.03907/) (2019)
- [Liquid Time-constant Networks](https://mlsanity.com/paper/2006.04439/) (2020)
- [Meta-Gradient Reinforcement Learning](https://mlsanity.com/paper/1805.09801/) (2018)
- [Deep Variational Reinforcement Learning for POMDPs](https://mlsanity.com/paper/1806.02426/) (2018)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Learning Long-Term Dependencies in Irregularly-Sampled Time Series](https://mlsanity.com/paper/2006.04418/) (2020)
- [Memory-based control with recurrent neural networks](https://mlsanity.com/paper/1512.04455/) (2015)
- [Recurrent Environment Simulators](https://mlsanity.com/paper/1704.02254/) (2017)
- [Memory-based Deep Reinforcement Learning for POMDPs](https://mlsanity.com/paper/2102.12344/) (2021)
- [Variational Recurrent Models for Solving Partially Observable Control Tasks](https://mlsanity.com/paper/1912.10703/) (2019)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Causal models for dynamical systems](https://mlsanity.com/paper/2001.06208/) (2020)

## Built on by (most cited first, 7 of 7)

- [Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains](https://mlsanity.com/paper/2508.00046/) (2025)
- [Biologically-Plausible Topology Improved Spiking Actor Network for Efficient Deep Reinforcement Learning](https://mlsanity.com/paper/2403.20163/) (2024)
- [Enhancing Solutions for Complex PDEs: Introducing Complementary Convolution and Equivariant Attention in Fourier Neural Operators](https://mlsanity.com/paper/2311.12902/) (2023)
- [Multiscale fusion enhanced spiking neural network for invasive BCI neural signal decoding](https://mlsanity.com/paper/2410.03533/) (2024)
- [A Convolution and Attention Based Encoder for Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2505.23857/) (2025)
- [Success in Humanoid Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2507.18883/) (2025)
- [Learning Causal States Under Partial Observability and Perturbation](https://mlsanity.com/paper/2512.00357/) (2025)

## Abstract (arXiv)

Neural ordinary differential equations (ODEs) are widely recognized as the standard for modeling physical mechanisms, which help to perform approximate inference in unknown physical or biological environments. In partially observable (PO) environments, how to infer unseen information from raw observations puzzled the agents. By using a recurrent policy with a compact context, context-based reinforcement learning provides a flexible way to extract unobservable information from historical transitions. To help the agent extract more dynamics-related information, we present a novel ODE-based recurrent model combines with model-free reinforcement learning (RL) framework to solve partially observable Markov decision processes (POMDPs). We experimentally demonstrate the efficacy of our methods across various PO continuous control and meta-RL tasks. Furthermore, our experiments illustrate that our method is robust against irregular observations, owing to the ability of ODEs to model irregularly-sampled time series.