mlsanityData to 14 Oct 2026

Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning

arXiv 2408.15368 ↗First posted 27 Aug 2024cs.LG cs.SY eess.SYcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 18 of its 28 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

18 of 28 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  2. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  3. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  6. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 10

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2017Dispersion for Data-Driven Algorithm Design, Online Learning, and Private Optimizationcited by 37
  5. 2019Learning Convex Optimization Control Policiescited by 31
  6. 2020CityLearn: Standardizing Research in Multi-Agent Reinforcement Learning for Demand Response and Urban Energy Managementcited by 27
  7. 2022Bounded-Regret MPC via Perturbation Analysis: Prediction Error, Constraints, and Nonlinearitycited by 16
  8. 2022Winning the CityLearn Challenge: Adaptive Optimization with Evolutionary Search under Trajectory-based Guidancecited by 7
  9. 2022On Solution Functions of Optimization: Universal Approximation and Covering Number Boundscited by 6
  10. 2021Exponential Decay of Sensitivity in Graph-Structured Nonlinear Programscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  2. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL
  3. 2024Adversarially Trained Weighted Actor-Critic for Safe Offline Reinforcement Learning
  4. 2021Offline RL Without Off-Policy Evaluation
  5. 2025Quasi-Newton Compatible Actor-Critic for Deterministic Policies
  6. 2019Behavior Regularized Offline Reinforcement Learning
  7. 2024AFU: Actor-Free critic Updates in off-policy RL for continuous control
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning