mlsanityData to 14 Oct 2026

Lifted Bellman Linear Programming for Offline Reinforcement Learning

arXiv 2609.24489 ↗First posted 21 Sep 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) typically trains a critic by minimizing a regression loss against bootstrapped value targets stabilized by target networks with exponential moving average (EMA) updates. Multi-step targets incorporate behavior-policy actions and therefore require off-policy correction. We instead impose in-sample Bellman optimality on the critic through inequality constraints. We formulate the Lifted Bellman Linear Program (LBLP), which lifts the linear programming characterization of Bellman optimality to the joint (Q,V) space so that every constraint involves only state-action pairs in the dataset. Its unique minimizer is the in-sample optimal pair, and constraints along K-step segments of dataset trajectories leave this minimizer unchanged for any rollout policy and horizon. Under deterministic dynamics, this minimizer lies between the best dataset return and the optimal value. Relaxing the constraints into hinge penalties recovers the same solution above a finite penalty coefficient in the tabular case. Approximate Lifted Bellman Unconstrained Minimization (ALBUM) implements this relaxation with neural networks and detaches the K-step rollout targets by stop gradient. Its objective contains no squared regression onto bootstrapped targets, so it can be trained without target networks or EMA updates. Under deterministic dynamics, the LBLP solution is a stationary point of the detached update under a coefficient condition independent of γ and K, and the inequality constraints allow discounted returns along dataset trajectories to serve as lower bounds without off-policy correction or action chunking. On OGBench, ALBUM uses a single critic with a Gaussian policy, matches the average performance of FQL, and is comparable to recent action-chunking methods, while using the fewest parameters and the least peak GPU memory among all compared methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 56 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

46 of 56 references.

Closest ancestors · 18

Same direction, newest first

  1. 2026Chunk-Guided Q-Learning14 Mar
  2. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RL9 Oct
  3. 2025Reinforcement Learning with Action Chunking10 Jul
  4. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?13 Jun
  6. 2024Learning from Sparse Offline Datasets via Conservative Density Estimation16 Jan
  7. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs22 May
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  9. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  10. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learning16 Feb
  11. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  12. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
6 more
  1. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability9 Feb
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 28

  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2021Convex optimizationcited by 2,515
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  10. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  11. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
16 more
  1. 2025Flow Q-Learningcited by 184
  2. 2018Self-Imitation Learningcited by 138
  3. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  4. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  5. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  6. 2023Score Regularized Policy Optimization through Diffusion Behaviorcited by 71
  7. 2020Logistic Q-Learningcited by 34
  8. 2019CAQL: Continuous Action Q-Learningcited by 23
  9. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  10. 2025Decoupled Q-Chunkingcited by 20
  11. 2021Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrationscited by 10
  12. 2017A Linearly Relaxed Approximate Linear Program for Markov Decision Processescited by 9
  13. 2018Stochastic Primal-Dual Q-Learningcited by 7
  14. 2023Convex Q Learning in a Stochastic Environment: Extended Versioncited by 5
  15. 2020Convex Q-Learning, Part 1: Deterministic Optimal Controlcited by 4
  16. 2025Analysis of approximate linear programming solution to Markov decision problem with log barrier functioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Stability and Generalization for Bellman Residuals
  2. 2024The Role of Inherent Bellman Error in Offline Reinforcement Learning with Linear Function Approximation
  3. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
  4. 2026Residuals-based Offline Reinforcement Learning
  5. 2024Computationally Efficient RL under Linear Bellman Completeness for Deterministic Dynamics
  6. 2026Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations
  7. 2025Bellman Calibration for V-Learning in Offline Reinforcement Learning
  8. 2022Learning Bellman Complete Representations for Offline Policy Evaluation