mlsanityData to 7 Oct 2026

TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning

arXiv 2604.00438 ↗First posted 1 Apr 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In-Context Reinforcement Learning (ICRL) enables Large Language Models (LLMs) to learn online from external rewards directly within the context window. However, a central challenge in ICRL is reward estimation, as models typically lack access to ground-truths during inference. To address this limitation, we propose Test-Time Rethinking for In-Context Reinforcement Learning (TR-ICRL), a novel ICRL framework designed for both reasoning and knowledge-intensive tasks. TR-ICRL operates by first retrieving the most relevant instances from an unlabeled evaluation set for a given query. During each ICRL iteration, LLM generates a set of candidate answers for every retrieved instance. Next, a pseudo-label is derived from this set through majority voting. This label then serves as a proxy to give reward messages and generate formative feedbacks, guiding LLM through iterative refinement. In the end, this synthesized contextual information is integrated with the original query to form a comprehensive prompt, with the answer determining through a final round of majority voting. TR-ICRL is evaluated on mainstream reasoning and knowledge-intensive tasks, where it demonstrates significant performance gains. Remarkably, TR-ICRL improves Qwen2.5-7B by 21.23% on average on MedQA and even 137.59% on AIME2024. Extensive ablation studies and analyses further validate the effectiveness and robustness of our approach. Our code is available at https://github.com/pangpang-xuan/TR_ICRL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 40 references that are in the map. First posted on arXiv 1 Apr 2026.

Builds on

24 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026In-Context Reinforcement Learning for Tool Use in Large Language Models9 Mar
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  3. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  5. 2024Free Process Rewards without Process Labels2 Dec

Other references · 19

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2024OpenAI o1 System Cardcited by 1,220
  9. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  10. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  12. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
7 more
  1. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  2. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  3. 2024OpenR: An Open Source Framework for Advanced Reasoning with Large Language Modelscited by 92
  4. 2025Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Aheadcited by 43
  5. 2025A Survey of In-Context Reinforcement Learningcited by 35
  6. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 29
  7. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025TTRL: Test-Time Reinforcement Learning
  2. 2026In-Context Reinforcement Learning under Non-Stationarity: A Survey
  3. 2025Scalable In-Context Q-Learning
  4. 2026Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
  5. 2025RoiRL: Efficient, Self-Supervised Reasoning with Offline Iterative Reinforcement Learning
  6. 2026LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards
  7. 2026Downstream-Aware Context Selection for Online In-Context Reinforcement Learning
  8. 2025ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models