mlsanityData to 14 Oct 2026

Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learning

arXiv 2609.29548 ↗First posted 26 Aug 2026cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Language-model advice can accelerate reinforcement learning, but calls are costly and returned actions may be stale or wrong. We formulate advice acquisition as a response-contingent metareasoning problem: before querying, the controller predicts possible parsed responses, evaluates the decision and declared continuation that would follow each response, and queries only when a lower confidence bound on predictive value exceeds the priced cost. Execution is governed separately by an action-specific certificate. Under explicit assumptions, certified advice is near-optimal, a wrapped learner inherits fallback regret only under intervention stability, and conservative allocation loses at most the declared query-value estimation error relative to a myopic oracle. On BabyAI, a proxy-calibrated controller with Qwen2.5-1.5B and 7B advisors improves GoToObj return over no querying by 0.029 +/- 0.016 and 0.030 +/- 0.015 across 20 seeds while reducing calls by more than 97% relative to always-query. GoToLocal is a null result. Exactly matched-call tests show an advantage over random placement only for the 1.5B advisor and no advantage over an equal-budget early schedule. Mondrian calibration improves decision-relevant empirical coverage from 0.47 to 0.85, still below the 0.90 target, while the formally covered radius is vacuous. The demonstrated benefit is therefore robust sparse advice volume on a useful task, not a proven per-state placement advantage.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 11 of its 19 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

11 of 19 references.

References · 11

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  3. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  4. 2016Deep Exploration via Bootstrapped DQNcited by 527
  5. 2017A Tutorial on Thompson Samplingcited by 433
  6. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  7. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  8. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  9. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  10. 2020FrugalML: How to Use ML Prediction APIs More Accurately and Cheaplycited by 33
  11. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximizationcited by 30

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  2. 2024Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Models
  3. 2026Learning Agentic Policy from Action Guidance
  4. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  5. 2026Training Language Models to Fail Gracefully
  6. 2026AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation
  7. 2026The Value Axis: Language Models Encode Whether They're on the Right Track
  8. 2025When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training