mlsanityData to 7 Oct 2026

General Flexible f-divergence for Challenging Offline RL Datasets with Low Stochasticity and Diverse Behavior Policies

arXiv 2602.11087 ↗First posted 11 Feb 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline RL algorithms aim to improve upon the behavior policy that produces the collected data while constraining the learned policy to be within the support of the dataset. However, practical offline datasets often contain examples with little diversity or limited exploration of the environment, and from multiple behavior policies with diverse expertise levels. Limited exploration can impair the offline RL algorithm's ability to estimate textitQ or textitV values, while constraining towards diverse behavior policies can be overly conservative. Such datasets call for a balance between the RL objective and behavior policy constraints. We first identify the connection between f-divergence and optimization constraint on the Bellman residual through a more general Linear Programming form for RL and the convex conjugate. Following this, we introduce the general flexible function formulation for the f-divergence to incorporate an adaptive constraint on algorithms' learning objectives based on the offline training dataset. Results from experiments on the MuJoCo, Fetch, and AdroitHand environments show the correctness of the proposed LP form and the potential of the flexible f-divergence in improving performance for learning from a challenging dataset when applied to a compatible constrained optimization algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 39 references that are in the map. First posted on arXiv 11 Feb 2026.

Builds on

21 of 39 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  2. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning16 Feb
  3. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  4. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 10

  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  3. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  6. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  8. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  9. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  10. 2024Neural Quasiprobabilistic Likelihood Ratio Estimation with Negatively Weighted Datacited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline RL Policies Should be Trained to be Adaptive
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2026f-Divergence Regularized RLHF: Two Tales of Sampling and Unified Analyses
  4. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
  5. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  6. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  7. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators
  8. 2021Instabilities of Offline RL with Pre-Trained Neural Representation