# Locality Sensitive Sparse Encoding for Learning World Models Online

arXiv: https://arxiv.org/abs/2401.13034 · HTML: https://mlsanity.com/paper/2401.13034/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 30 of its 58 references that are in the map.
- First posted on arXiv 23 Jan 2024.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 30)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Online Learning: A Modern Introduction Using Convex Optimization](https://mlsanity.com/paper/1912.13213/) (2019)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [On Tiny Episodic Memories in Continual Learning](https://mlsanity.com/paper/1902.10486/) (2019)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [Online Learning: A Comprehensive Survey](https://mlsanity.com/paper/1802.02871/) (2018)
- [The Power of Depth for Feedforward Neural Networks](https://mlsanity.com/paper/1512.03965/) (2015)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Towards Generalization and Simplicity in Continuous Control](https://mlsanity.com/paper/1703.02660/) (2017)
- [ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection](https://mlsanity.com/paper/2205.14922/) (2022)
- [Offline Reinforcement Learning with Reverse Model-based Imagination](https://mlsanity.com/paper/2110.00188/) (2021)
- [Self-Correcting Models for Model-Based Reinforcement Learning](https://mlsanity.com/paper/1612.06018/) (2016)
- [Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Models](https://mlsanity.com/paper/2006.04363/) (2020)
- [The Utility of Sparse Representations for Control in Reinforcement Learning](https://mlsanity.com/paper/1811.06626/) (2018)
- [Learning Dynamics Models for Model Predictive Agents](https://mlsanity.com/paper/2109.14311/) (2021)
- [The Alberta Plan for AI Research](https://mlsanity.com/paper/2208.11173/) (2022)
- [Uniform regret bounds over Rd for the sequential linear regression problem with the square loss](https://mlsanity.com/paper/1805.11386/) (2018)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [Efficient Offline Policy Optimization with a Learned Model](https://mlsanity.com/paper/2210.05980/) (2022)
- [Hill Climbing on Value Estimates for Search-control in Dyna](https://mlsanity.com/paper/1906.07791/) (2019)
- [Learning Sparse Representations Incrementally in Deep Reinforcement Learning](https://mlsanity.com/paper/1912.04002/) (2019)

## Built on by (most cited first, 12 of 12)

- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [A Survey of Continual Reinforcement Learning](https://mlsanity.com/paper/2506.21872/) (2025)
- [Sample-Efficient Alignment for LLMs](https://mlsanity.com/paper/2411.01493/) (2024)
- [Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models](https://mlsanity.com/paper/2406.18868/) (2024)
- [F-OAL: Forward-only Online Analytic Learning with Fast Training and Low Memory Footprint in Class Incremental Learning](https://mlsanity.com/paper/2403.15751/) (2024)
- [AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Models](https://mlsanity.com/paper/2405.16240/) (2024)
- [Continual Reinforcement Learning by Planning with Online World Models](https://mlsanity.com/paper/2507.09177/) (2025)
- [Knowledge Retention for Continual Model-Based Reinforcement Learning](https://mlsanity.com/paper/2503.04256/) (2025)
- [Online Analytic Exemplar-Free Continual Learning with Large Models for Imbalanced Autonomous Driving Task](https://mlsanity.com/paper/2405.17779/) (2024)
- [AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Data](https://mlsanity.com/paper/2505.12245/) (2025)
- [Lifelong Learning of Video Diffusion Models From a Single Video Stream](https://mlsanity.com/paper/2406.04814/) (2024)
- [FedHiP: Heterogeneity-Invariant Personalized Federated Learning Through Closed-Form Solutions](https://mlsanity.com/paper/2508.04470/) (2025)

## Abstract (arXiv)

Acquiring an accurate world model online for model-based reinforcement learning (MBRL) is challenging due to data nonstationarity, which typically causes catastrophic forgetting for neural networks (NNs). From the online learning perspective, a Follow-The-Leader (FTL) world model is desirable, which optimally fits all previous experiences at each round. Unfortunately, NN-based models need re-training on all accumulated data at every interaction step to achieve FTL, which is computationally expensive for lifelong agents. In this paper, we revisit models that can achieve FTL with incremental updates. Specifically, our world model is a linear regression model supported by nonlinear random features. The linear part ensures efficient FTL update while the nonlinear random feature empowers the fitting of complex environments. To best trade off model capacity and computation efficiency, we introduce a locality sensitive sparse encoding, which allows us to conduct efficient sparse updates even with very high dimensional nonlinear features. We validate the representation power of our encoding and verify that it allows efficient online learning under data covariate shift. We also show, in the Dyna MBRL setting, that our world models learned online using a single pass of trajectory data either surpass or match the performance of deep world models trained with replay and other continual learning methods.