mlsanityData to 7 Oct 2026

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

arXiv 2604.22873 ↗First posted 23 Apr 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) can learn effective policies from fixed datasets, but deployment objectives may change after training, and in many applications the trained actor cannot be retrained because of data, cost, or governance constraints. We study deployment-time adaptation for frozen offline actors using Product-of-Experts (PoE) composition with a goal-conditioned prior. Our main practical finding is graceful degradation rather than universal performance gain: under degraded or random priors, precision-weighted composition remains anchored to the frozen actor, while additive and prior-only adaptation collapse, and a KL-budget selector often recovers a near-oracle operating point. We also make explicit a closed-form identity in the frozen-actor setting: for diagonal-Gaussian actors and priors, PoE with coefficient alpha yields the same deterministic policy as KL-regularized adaptation with beta = alpha / (1 - alpha), with posterior covariances differing only by a global scalar factor. Empirically, across four D4RL environments (3,900 MuJoCo episodes), we observe a 4/5/3 HELP/FROZEN/HURT split. Extending the analysis to six harder cells and two AntMaze diagnostics reveals an actor-competence ceiling: medium-expert remains HURT in all 9 cells at every tested alpha, while AntMaze with a behavior-cloned frozen actor yields zero success for all composition rules. Overall, PoE and KL-regularized adaptation are best viewed as a single actor-anchored safety mechanism for deployment-time steering.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 20 of its 25 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

20 of 25 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Epistemic Robust Offline Reinforcement Learning8 Apr
  2. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning9 May
  3. 2023CROP: Conservative Reward for Model-based Offline Policy Optimization26 Oct
  4. 2023Conservative State Value Estimation for Offline Reinforcement Learning14 Feb
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 13

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  6. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  7. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  8. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  9. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  10. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  11. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  12. 2018Composable Deep Reinforcement Learning for Robotic Manipulationcited by 70
1 more
  1. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
  2. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  3. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  6. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  7. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  8. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning