# From Passive Reuse to Active Reasoning: Grounding Large Language Models for Neuro-Symbolic Experience Replay

arXiv: https://arxiv.org/abs/2605.09419 · HTML: https://mlsanity.com/paper/2605.09419/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 10 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Pre-Trained Language Models for Interactive Decision-Making](https://mlsanity.com/paper/2202.01771/) (2022)
- [Revisiting Fundamentals of Experience Replay](https://mlsanity.com/paper/2007.06700/) (2020)
- [Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning](https://mlsanity.com/paper/2408.10075/) (2024)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs](https://mlsanity.com/paper/2404.08555/) (2024)
- [Partially Observable Markov Decision Processes in Robotics: A Survey](https://mlsanity.com/paper/2209.10342/) (2022)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [Autoformalizing Natural Language to First-Order Logic: A Case Study in Logical Fallacy Detection](https://mlsanity.com/paper/2405.02318/) (2024)
- [Efficient Symbolic Policy Learning with Differentiable Symbolic Expression](https://mlsanity.com/paper/2311.02104/) (2023)
- [ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Design](https://mlsanity.com/paper/2306.08888/) (2023)
- [Temporal-Difference Learning Using Distributed Error Signals](https://mlsanity.com/paper/2411.03604/) (2024)
- [LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition](https://mlsanity.com/paper/2509.01337/) (2025)
- [Online Symbolic Regression with Informative Query](https://mlsanity.com/paper/2302.10539/) (2023)
- [Neurosymbolic Inference On Foundation Models For Remote Sensing Text-to-image Retrieval With Complex Queries](https://mlsanity.com/paper/2512.14102/) (2025)

## Abstract (arXiv)

While experience replay is essential for data efficiency in reinforcement learning (RL), standard methods treat the replay buffer as a passive memory system, prioritizing samples based on numerical prediction errors rather than their semantic significance. This approach stands in contrast to human learning, which accelerates mastery by actively abstracting fragmented experiences into behavioral rules. To bridge this gap, we propose Neuro-Symbolic Experience Replay (NSER), a framework that transforms experience replay from a passive sample reuse mechanism into an active engine for knowledge construction. Specifically, NSER addresses the incompatibility between linguistic reasoning and numerical optimization through a novel neuro-symbolic grounding pipeline. It leverages Large Language Models (LLMs) in a zero-shot manner to induce candidate behavioral rules from accumulated trajectories, grounds these insights into differentiable first-order logic representations, and utilizes the resulting symbolic structures to dynamically reweight the replay distribution. By allowing abstract knowledge to directly shape policy optimization, NSER achieves consistent superior sample efficiency and convergence speed across reactive, rule-based, and procedural benchmarks.