mlsanityData to 7 Oct 2026

Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

arXiv 2608.03545 ↗First posted 4 Aug 2026cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 41 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

28 of 41 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026SARL: Label-Free Reinforcement Learning by Rewarding Reasoning Topology30 Mar
  2. 2026What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time20 Mar
  3. 2026CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution18 Mar
  4. 2025Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning17 Dec
  5. 2025Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement20 Nov
  6. 2025StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason3 Jul
  7. 2025Learning to Reason without External Rewards26 May
  8. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  7. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  8. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  9. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  10. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  11. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  12. 2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
5 more
  1. 2025Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Trainingcited by 29
  2. 2025RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Taskscited by 12
  3. 2025Self-Harmony: Learning to Harmonize Self-Supervision and Self-Play in Test-Time Reinforcement Learningcited by 9
  4. 2026ECHO: Entropy-Confidence Hybrid Optimization for Test-Time Reinforcement Learningcited by 4
  5. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matchingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TTPO: Test-Time Policy Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025TTRL: Test-Time Reinforcement Learning
  2. 2026Tool Verification for Test-Time Reinforcement Learning
  3. 2026Detecting and Mitigating the Correct-Answer Extinction Window in Test-Time Reinforcement Learning with Majority Voting
  4. 2026Exploiting Verification-Generation Gap: Test-Time Reinforcement Learning with Confidence-Conditioned Verification
  5. 2026Small RL Controller, Large Language Model: RL-Guided Adaptive Sampling for Test-Time Scaling
  6. 2026Dual Consensus: Escaping from Spurious Majority in Unsupervised RLVR via Two-Stage Vote Mechanism
  7. 2026EchoRL: Reinforcement Learning via Rollout Echoing
  8. 2026ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying