mlsanityData to 14 Oct 2026

Symmetric Behavior Regularized Policy Optimization

arXiv 2508.04225 ↗First posted 6 Aug 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Behavior Regularized Policy Optimization (BRPO) leverages asymmetric divergence regularization to mitigate distribution shift in offline reinforcement learning. This paper is the first to study the open question of symmetric BRPO. Using didactic examples, we show that symmetric regularization can outperform asymmetric regularization in addressing one-sided bias, near-boundary policy updates, and projection geometry consistency. However, symmetric divergences do not fit BRPO naturally: they do not permit a closed-form solution when used as regularizers, and can lead to numerical instability when used as optimization objectives. We first introduce a universal BRPO framework using an infinite series of Pearson-Vajda divergences to represent any f-divergence, which includes both symmetric and asymmetric divergences. We use a finite-series approximation to obtain the following results for symmetric BRPO: (1) a closed-form optimal policy expression; (2) a numerically stable optimization surrogate; and (3) a tight upper bound on the approximation quality. On the D4RL benchmark and in didactic examples, we show that the proposed method achieves consistently strong results and is robust to the number of terms in the approximation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 51 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

34 of 51 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Behavior Preference Regression for Offline Reinforcement Learning2 Mar
  2. 2025Fat-to-Thin Policy Optimization: Offline RL with Sparse Policies24 Jan
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update1 Feb
  4. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning9 Jun
  5. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  6. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  7. 2023Behavior Proximal Policy Optimization22 Feb
  8. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  9. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
2 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  9. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  10. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  11. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  12. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
8 more
  1. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  2. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 41
  3. 2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
  4. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimizationcited by 19
  5. 2021A general class of surrogate functions for stable and efficient reinforcement learningcited by 18
  6. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  7. 2020Taylor Expansion Policy Optimizationcited by 9
  8. 2023Quasi-optimal Reinforcement Learning with Continuous Actionscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning
  2. 2021Offline Reinforcement Learning with Soft Behavior Regularization
  3. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization
  5. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
  6. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning
  7. 2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularization
  8. 2025Divergence-Augmented Policy Optimization