Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Exploration
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
While offline reinforcement learning provides reliable policies for real-world deployment, its inherent pessimism severely restricts an agent's ability to explore and collect novel data online. Drawing inspiration from safe reinforcement learning, exploring near the boundary of regions well covered by the offline dataset and reliably modeled by the simulator allows an agent to take manageable risks--venturing into informative but moderate-uncertainty states while remaining close enough to familiar regions for safe recovery. However, naively rewarding this boundary-seeking behavior can lead to a degenerate parking behavior, where the agent simply stops once it reaches the frontier. To solve this, we propose a novel vector-field reward shaping paradigm designed to induce continuous, safe boundary exploration for non-adaptive deployed policies. Operating on an uncertainty oracle trained from offline data, our reward combines two complementary components: a gradient-alignment term that attracts the agent toward a target uncertainty level, and a rotational-flow term that promotes motion along the local tangent plane of the uncertainty manifold. Through theoretical analysis, we show that this reward structure naturally induces sustained exploratory behavior along the boundary while preventing degenerate solutions. Empirically, by integrating our proposed reward shaping with Soft Actor-Critic on a 2D continuous navigation task, we validate that agents successfully traverse uncertainty boundaries while balancing safe, informative data collection with primary task completion.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 26 of its 31 references that are in the map. First posted on arXiv 18 Mar 2026.
Builds on
26 of 31 references.
Closest ancestors · 5
Same direction, newest first
- 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
Other references · 21
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
- 2018Exploration by Random Network Distillationcited by 966
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
- 2018Visual Reinforcement Learning with Imagined Goalscited by 248
- 2018Provably Efficient Maximum Entropy Explorationcited by 240
- 2019Efficient Exploration via State Marginal Matchingcited by 186
- 2016Bayesian Optimization with Safety Constraints: Safe and Automatic Parameter Tuning in Roboticscited by 134
- 2020Stochastic Bandits with Linear Constraintscited by 68
- 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
- 2017EX2: Exploration with Exemplar Models for Deep Reinforcement Learningcited by 42
- 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 29
9 more
- 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 24
- 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
- 2023A Near-Optimal Algorithm for Safe Reinforcement Learning Under Instantaneous Hard Constraintscited by 15
- 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
- 2024Contextual Bandits with Stage-wise Constraintscited by 12
- 2023Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Datacited by 11
- 2025Provably Efficient RL for Linear MDPs under Instantaneous Safety Constraints in Non-Convex Feature Spacescited by 6
- 2025Provably Efficient RL under Episode-Wise Safety in Constrained MDPs with Linear Function Approximationcited by 4
- 2016A guiding vector field algorithm for path following control of nonholonomic mobile robotscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning
- 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning
- 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning
- 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning
- 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
- 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
- 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching
- 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning