mlsanityData to 7 Oct 2026

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning

arXiv 2609.33166 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

In-context reinforcement learning (ICRL) enables large language model agents to adapt to new environments using their interaction history without updating model parameters. However, repeatedly conditioning on growing histories can lead to substantial token cost. We propose a bounded-history context-management framework that predicts the task-dependent downstream effect of removing historical interactions to guide history selection and determine a decision-dependent context budget. Formally, our framework uses the full rolling history as a reference. The predictor evaluates removal effects, defines a deletion ordering, and applies a shared selection criterion to determine how much history to retain at each decision. We evaluate the method in closed-loop SUMO driving under held-out in-distribution, unseen-domain, and unseen-route settings, and in ScienceWorld under a continual ICRL protocol. Relative to a baseline using the full context, our method reduces total token usage by 25.7%, 25.8%, and 23.2% across the three driving settings while maintaining comparable closed-loop driving performance. In ScienceWorld, it reduces total token usage by 52.1% compared to full context and uses 30.2% and 37.8% fewer tokens than the Recent and Similarity baselines, respectively, while maintaining performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 32 of its 35 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

32 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2026Agent-Omit: Adaptive Context Omission for Efficient LLM Agents4 Feb

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  3. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  4. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  6. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  7. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  8. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 226
  9. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  10. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  11. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  12. 2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
19 more
  1. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 126
  2. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  3. 2023ProAgent: Building Proactive Cooperative Agents with Large Language Modelscited by 97
  4. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  5. 2025Memory as Action: Autonomous Context Curation for Long-Horizon Agentic Taskscited by 62
  6. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 57
  7. 2023Skill-Based Few-Shot Selection for In-Context Learningcited by 46
  8. 2024LimSim++: A Closed-Loop Platform for Deploying Multimodal LLMs in Autonomous Drivingcited by 32
  9. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  10. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thoughtcited by 25
  11. 2023RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learningcited by 24
  12. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 20
  13. 2025Sample Efficient Demonstration Selection for In-Context Learningcited by 12
  14. 2024SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Modelscited by 10
  15. 2024Demonstration Selection for In-Context Learning via Reinforcement Learningcited by 7
  16. 2025Influence Guided Context Selection for Effective Retrieval-Augmented Generationcited by 6
  17. 2025Learning to Select In-Context Demonstration Preferred by Large Language Modelcited by 5
  18. 2025Filtering Learning Histories Enhances In-Context Reinforcement Learningcited by 3
  19. 2025Prompt-Driven Domain Adaptation for End-to-End Autonomous Driving via In-Context RLcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Survey
  2. 2026Escaping the Context Bottleneck: Active Context Curation for LLM Agents via Reinforcement Learning
  3. 2026TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning
  4. 2024Can large language models explore in-context?
  5. 2026ContextBudget: Budget-Aware Context Management for Long-Horizon Search Agents
  6. 2025Scalable In-Context Q-Learning
  7. 2026Considering Context: When World Models Need Context Encoders
  8. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning