mlsanityData to 7 Oct 2026

When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning

arXiv 2605.09109 ↗First posted 9 May 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Many continuous-control problems ship with a competent but suboptimal controller (a tuned PID, a hand-designed gait). A growing family of methods uses such controllers as queryable experts during RL, but each method has been proposed in isolation, on a different benchmark, without imperfect-expert testing. We harmonize the comparison on a shared SAC backbone, common HPO and evaluation protocols, 100/50 seeds per (env, method), and a degradation sweep over expert undertuning, action bias, and observation noise. The comparison surfaces three failure modes single-paper evaluations miss: (F1) a critic blind spot under argmax-plus-bootstrap that drags IBRL below no-expert SAC on experts close to the no-expert-RL ceiling (RL-near-ceiling, distinct from the absolute physical ceiling); (F2) residual saturation on far-from-optimal experts; and (F3) warm-start buffer poisoning that collapses training-time-handoff methods under deployment-time expert undertuning. No single method dominates: each wins on one task-structure regime and fails predictably elsewhere; on RL-near-ceiling experts (FourTank, GlassFurnace) no query-time method clears the expert within our 1M-step budget, leaving open whether this is a fundamental wall or a budget effect. We convert the spread into a testable decision rule keyed on three pre-training observables (expert quality, task termination, perturbation type). The benchmark, taxonomy, and decision rule are the primary contribution; we additionally describe EDGE, a softmax-over-ensemble-LCB design point used to demonstrate that both axes the taxonomy points to (gate form, scoring rule) are individually exploitable.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 9 May 2026.

Builds on

23 of 36 references.

References · 23

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018DeepMind Control Suitecited by 935
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2016Deep Exploration via Bootstrapped DQNcited by 518
  10. 2017A Tutorial on Thompson Samplingcited by 420
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
11 more
  1. 2013Learning to Optimize Via Posterior Samplingcited by 253
  2. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  3. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  4. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  5. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
  6. 2016Conservative Banditscited by 58
  7. 2016Conservative Contextual Linear Banditscited by 56
  8. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  9. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  10. 2024PC-Gym: Benchmark Environments For Process Control Problemscited by 8
  11. 2025Solving Robotics Tasks with Prior Demonstration via Exploration-Efficient Deep Reinforcement Learningcited by 1

Built on by

Not cited yet.