mlsanityData to 7 Oct 2026

Quantifying Memory Use in Reinforcement Learning with Temporal Range

arXiv 2512.06204 ↗First posted 5 Dec 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

How much does a trained RL policy actually use its past observations? We propose Temporal Range, a model-agnostic metric that treats first-order sensitivities of multiple vector outputs across a temporal window to the input sequence as a temporal influence profile and summarizes it by the magnitude-weighted average lag. Temporal Range is computed via reverse-mode automatic differentiation from the Jacobian blocks partial ys/partial xt∈ℝc× d averaged over final timesteps s∈t+1,dots,T and is well-characterized in the linear setting by a small set of natural axioms. Across diagnostic and control tasks (POPGym; flicker/occlusion; Copy-k) and architectures (MLPs, RNNs, SSMs), Temporal Range (i) remains small in fully observed control, (ii) scales with the task's ground-truth lag in Copy-k, and (iii) aligns with the minimum history window required for near-optimal return as confirmed by window ablations. We also report Temporal Range for a compact Long Expressive Memory (LEM) policy trained on the task, using it as a proxy readout of task-level memory. Our axiomatic treatment draws on recent work on range measures, specialized here to temporal lag and extended to vector-valued outputs in the RL setting. Temporal Range thus offers a practical per-sequence readout of memory dependence for comparing agents and environments and for selecting the shortest sufficient context.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

27 of 38 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2018Sanity Checks for Saliency Mapscited by 1,149
  12. 2017SmoothGrad: removing noise by adding noisecited by 1,097
15 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  2. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  4. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  5. 2017Visualizing and Understanding Atari Agentscited by 141
  6. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  7. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  8. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  9. 2018Can recurrent neural networks warp time?cited by 53
  10. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  11. 2021Long Expressive Memory for Sequence Modelingcited by 41
  12. 2022Provable Reinforcement Learning with a Short-Term Memorycited by 40
  13. 2024Oscillatory State-Space Modelscited by 35
  14. 2025On Measuring Long-Range Interactions in Graph Neural Networkscited by 9
  15. 2022Active Inference and Reinforcement Learning: A unified inference on continuous state and action spaces under partial observabilitycited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
  2. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
  3. 2026Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
  4. 2025Memory Allocation in Resource-Constrained Reinforcement Learning
  5. 2025TempR1: Improving Temporal Understanding of MLLMs via Temporal-Aware Multi-Task Reinforcement Learning
  6. 2026Decision Titan: Test-Time Training for Long-Term Memory in Offline Reinforcement Learning
  7. 2026TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning
  8. 2026Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learning