mlsanityData to 7 Oct 2026

Decision Titan: Test-Time Training for Long-Term Memory in Offline Reinforcement Learning

arXiv 2610.01513 ↗First posted 1 Oct 2026cs.AI cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Long-term dependencies remain a major challenge for sequential decision-making in the field of AI: RNNs suffer from vanishing gradients and the limited expressivity of vector-based hidden states, whilst Transformer-based models are limited by the quadratic scaling of attention. Recent work has proposed tackling this problem with the Test-Time Training (TTT) framework, which stores episodic memories in the parameters of a neural network through gradient descent at both train and test-time. This approach has seen success in the domain of Natural Language Processing, however, to the best of our knowledge it has not yet been applied to the domain of Reinforcement Learning (RL), nor has there been a study analysing how this memory practically functions. In this paper, we study the potential of the TTT framework for offline RL by augmenting a Decision Transformer with TTT layers, dubbed the Decision Titan. We analyse performance and properties of the model in the X-Maze environment, an extension of T-Maze designed to test sequential memory, and investigate how the memory mechanism learns by visualising gate values over time. Our key findings are that Decision Titan can learn long-term dependencies with ranges 20x longer than the context window, generalises to lengths 1.7x the training data, but crucially temporal generalisation depends on the time embeddings used, and the ability to learn long-term dependencies depends on how the relevant information is encoded.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 27 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

22 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokens14 Sep
  2. 2023Recurrent Action Transformer with Memory15 Jun
  3. 2022Multi-Game Decision Transformers30 May
  4. 2022Online Decision Transformer11 Feb
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  6. 2018Self-Attention with Relative Position Representationscited by 1,112
  7. 2020Efficient Transformers: A Surveycited by 784
  8. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  9. 2024Titans: Learning to Memorize at Test Timecited by 303
  10. 2025Test-Time Training Done Rightcited by 121
  11. 2019Large Memory Layers with Product Keyscited by 120
  12. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
5 more
  1. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56
  2. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  3. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  4. 2026Fast-weight Product Key Memorycited by 11
  5. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TTRL: Test-Time Reinforcement Learning
  2. 2025Test-time Offline Reinforcement Learning on Goal-related Experience
  3. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
  4. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought
  5. 2025Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
  6. 2026Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning
  7. 2021Offline Reinforcement Learning with Value-based Episodic Memory
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Making