mlsanityData to 14 Oct 2026

Provable Transformer Context Compression for In-Context Reinforcement Learning

arXiv 2610.20004 ↗First posted 14 Oct 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations also fits: RL theory and sample complexity

Abstract (arXiv)

Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context sizes that grow with the number of sampled transitions, leading to increasing memory costs in large-scale problems. This raises a fundamental scalability question: how much context is intrinsically necessary to carry out such learning? We answer this question for fixed-policy TD evaluation. We show that, with a shared linear decoder, the minimum context dimension required to exactly reproduce the first L TD iterates is the Krylov dimension induced by the sampled transitions. When the available context dimension m falls below this threshold, exact reproduction is no longer possible. In this regime, we characterize the optimal approximation error and show that it decays as Theta(γmmax1,1/(m(1-γ))) where γ is the discounting factor. Moving from representation complexity to actual memory cost, we characterize the optimal retained memory for exact continuous representations as Theta(nkL) and show experimentally that on a sampled task whose exact rank is full but whose numerical rank is small, a context accurate to 10-2 is 88.7% smaller than the sparse sufficient statistics of TD. Finally, we establish the learnability of the shared executor through pretraining by proving global convergence of an auxiliary objective over effective attention parameters under suitable task diversity. Experiments across diverse TD evaluation problems validate parameter recovery and the predicted context--accuracy tradeoff.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 52 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

27 of 52 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning8 May
  2. 2026Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement7 May
  3. 2025Towards Provable Emergence of In-Context Reinforcement Learning22 Sep
  4. 2025A Survey of In-Context Reinforcement Learning11 Feb

Other references · 23

  1. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  5. 2019Deep Equilibrium Modelscited by 615
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  7. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  8. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  9. 2023Adapting Language Models to Compress Contextscited by 356
  10. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  11. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  12. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
11 more
  1. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  2. 2023Looped Transformers are Better at Learning Learning Algorithmscited by 149
  3. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  4. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  5. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  6. 2024Long Context Compression with Activation Beaconcited by 104
  7. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  8. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  9. 2024On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encodingcited by 36
  10. 2025Optimal Dynamic Regret by Transformers for Non-Stationary Reinforcement Learningcited by 2
  11. 2026Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thoughtcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  2. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Space
  3. 2024Statistical Analysis of Policy Space Compression Problem
  4. 2026Scalable In-Context Reinforcement Learning with Recurrent Algorithm Distillation
  5. 2026Statistical Convergence of Transformer Encoder-Accelerated Robust Reinforcement Learning
  6. 2025Filtering Learning Histories Enhances In-Context Reinforcement Learning
  7. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
  8. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RL