From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning

arXiv 2609.30391 ↗First posted 24 Sep 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Existing in-context reinforcement learning methods mainly pretrain Transformers with supervised behavior-prediction objectives. This enables task inference from context, but makes the learned policy strongly depend on the quality of offline actions: when trajectories are weak or suboptimal, imitation itself becomes a biased learning signal. We propose Q-Target Pretrained Transformers (QTPT), which keeps the context-conditioned Transformer architecture but replaces behavior cloning with a Bellman-style Q-target objective. QTPT therefore learns to use rewards and transitions in the context to estimate action values, rather than simply imitating the behavior policy. We theoretically analyze QTPT in stochastic linear bandits and finite-horizon MDPs, showing stronger robustness to data quality than supervised pretraining. Empirically, QTPT improves over supervised behavior prediction on controlled RL benchmarks with random or suboptimal data, and we examine extensions to D4RL Kitchen and AntMaze. Supplementary experiments evaluate backbone robustness, meta-RL comparisons, task-coherent context, and unsupported-action value overestimation. These comparisons distinguish the benefits of Q-target pretraining from the remaining limitations of offline coverage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 54 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

41 of 54 references.

Closest ancestors · 2

Same direction, newest first

Other references · 39

2021Training Verifiers to Solve Math Word Problemscited by 10,109
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
2022Transformers learn in-context by gradient descentcited by 732
2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
2019Behavior Regularized Offline Reinforcement Learningcited by 662

27 more

2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
2018Is Q-learning Provably Efficient?cited by 487
2021MetaICL: Learning to Learn In Contextcited by 454
2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
2020Is Pessimism Provably Efficient for Offline RL?cited by 364
2023Trained Transformers Learn Linear Models In-Contextcited by 337
2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
2020Q-learning with Logarithmic Regretcited by 60
2021Near-Optimal Offline Reinforcement Learning via Double Variance Reductioncited by 56
2023Behavior Proximal Policy Optimizationcited by 43
2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 29
2022On Instance-Dependent Bounds for Offline Reinforcement Learning with Linear Function Approximationcited by 22
2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way