mlsanityData to 14 Oct 2026

In-Context Compositional Q-Learning for Offline Reinforcement Learning

arXiv 2509.24067 ↗First posted 28 Sep 2025cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Accurate estimation of the Q-function is a central challenge in offline reinforcement learning. However, existing approaches often rely on a shared global Q-function, which is inadequate for capturing the compositional structure of tasks that consist of diverse subtasks. We propose In-context Compositional Q-Learning (ICQL), an offline RL framework that formulates Q-learning as a contextual inference problem and uses linear Transformers to adaptively infer local Q-functions from retrieved transitions without explicit subtask labels. Theoretically, we show that, under two assumptions -- linear approximability of the local Q-function and accurate inference of weights from retrieved context -- ICQL achieves a bounded approximation error for the Q-function and enables near-optimal policy extraction. Empirically, ICQL substantially improves performance in offline settings, achieving gains of up to 16.4% on kitchen tasks and up to 8.8% and 6.3% on MuJoCo and Adroit tasks, respectively. These results highlight the underexplored potential of in-context learning for robust and compositional value estimation and establish ICQL as a principled and effective framework for offline RL.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 65 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

43 of 65 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Reinforcement Learning with Action Chunking10 Jul
  2. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 33

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2022A Generalist Agentcited by 808
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  10. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  11. 2016The Option-Critic Architecturecited by 448
  12. 2022Transformers are Sample-Efficient World Modelscited by 328
21 more
  1. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  2. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  3. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  4. 2020Reinforcement Learning for Combinatorial Optimization: A Surveycited by 236
  5. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 229
  6. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  7. 2025Flow Q-Learningcited by 184
  8. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  9. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  10. 2023Linear attention is (maybe) all you need (to understand transformer optimization)cited by 98
  11. 2020OR-Gym: A Reinforcement Learning Library for Operations Research Problemscited by 51
  12. 2020Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedupcited by 25
  13. 2021Efficient Local Planning with Linear Function Approximationcited by 23
  14. 2024Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learningcited by 23
  15. 2020SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft IIcited by 22
  16. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  17. 2025PIKE-RAG: sPecIalized KnowledgE and Rationale Augmented Generationcited by 18
  18. 2020Finite-sample Analysis of Greedy-GQ with Linear Function Approximation under Markovian Noisecited by 17
  19. 2024Pretraining Decision Transformers with Reward Prediction for In-Context Multi-task Structured Bandit Learningcited by 10
  20. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
  21. 2024Hierarchical Universal Value Function Approximatorscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Yes, Q-learning Helps Offline In-Context RL
  2. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  3. 2025Scalable In-Context Q-Learning
  4. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  5. 2023Contextual Conservative Q-Learning for Offline Reinforcement Learning
  6. 2025Generalisation in Multitask Fitted Q-Iteration and Offline Q-learning
  7. 2024Strategically Conservative Q-Learning
  8. 2025BiCQL-ML: A Bi-Level Conservative Q-Learning Framework for Maximum Likelihood Inverse Reinforcement Learning