Context Value Informed In-Context Reinforcement Learning

arXiv 2509.23209 ↗First posted 27 Sep 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

In-context reinforcement learning (ICRL) has emerged as an effective paradigm for test time adaptation to unseen tasks without parameter updates. However, existing ICRL methods can exhibit brittle and unstable adaptions, and the mechanisms underlying such adaptions remain poorly understood. We provide a policy-gradient view of ICRL and argue that relying on trajectory-level return feedback can lead to high-variance updates when test time rollout is limited. Motivated by the actor--critic principle, we propose Context Value Informed ICRL (CV-ICRL), which equips the in-context policy with an explicit context value: the expected discounted return conditioned on the current state and accumulated interaction context. CV-ICRL trains a value head and writes its prediction back into the context as a value token, enabling TD-style return targets for lower-variance policy updates. Experiments on the Dark Room, Minigrid, and Procgen testbeds show that CV-ICRL substantially improves the stability of test time adaptation and achieves higher returns across tasks and environments. Code and data are available at https://github.com/Bluixe/CV-ICRL .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 21 of its 22 references that are in the map. First posted on arXiv 27 Sep 2025.

Builds on

21 of 22 references.

References · 21

2020Language Models are Few-Shot Learnerscited by 32,419
2017Proximal Policy Optimization Algorithmscited by 15,215
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2018Quantifying Generalization in Reinforcement Learningcited by 352
2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 146
2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
2024Can large language models explore in-context?cited by 76
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41

9 more

2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
2025Training a Generally Curious Agentcited by 31
2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 27
2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 26
2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
2023Bridging the Sim-to-Real Gap from the Information Bottleneck Perspectivecited by 10
2025Yes, Q-learning Helps Offline In-Context RLcited by 10
2025Scalable In-Context Q-Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way