# Off-Policy Average Reward Actor-Critic with Deterministic Policy Search

arXiv: https://arxiv.org/abs/2305.12239 · HTML: https://mlsanity.com/paper/2305.12239/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 18 references that are in the map.
- First posted on arXiv 20 May 2023.

## Builds on (most cited first, 3 of 3)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Examining average and discounted reward optimality criteria in reinforcement learning](https://mlsanity.com/paper/2107.01348/) (2021)

## Built on by (most cited first, 10 of 10)

- [Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning](https://mlsanity.com/paper/2405.19909/) (2024)
- [Asymptotic and Finite Sample Analysis of Nonexpansive Stochastic Approximations with Markovian Noise](https://mlsanity.com/paper/2409.19546/) (2024)
- [Average-Reward Soft Actor-Critic](https://mlsanity.com/paper/2501.09080/) (2025)
- [NeoRL: Efficient Exploration for Nonepisodic RL](https://mlsanity.com/paper/2406.01175/) (2024)
- [Average Reward Reinforcement Learning for Wireless Radio Resource Management](https://mlsanity.com/paper/2501.06700/) (2025)
- [When Do Off-Policy and On-Policy Policy Gradient Methods Align?](https://mlsanity.com/paper/2402.12034/) (2024)
- [Towards Optimal Offline Reinforcement Learning](https://mlsanity.com/paper/2503.12283/) (2025)
- [Almost Sure Convergence of Differential Temporal Difference Learning for Average Reward Markov Decision Processes](https://mlsanity.com/paper/2602.16629/) (2026)
- [An Empirical Study of Deep Reinforcement Learning in Continuing Tasks](https://mlsanity.com/paper/2501.06937/) (2025)
- [Off-OAB: Off-Policy Policy Gradient Method with Optimal Action-Dependent Baseline](https://mlsanity.com/paper/2405.02572/) (2024)

## Abstract (arXiv)

The average reward criterion is relatively less studied as most existing works in the Reinforcement Learning literature consider the discounted reward criterion. There are few recent works that present on-policy average reward actor-critic algorithms, but average reward off-policy actor-critic is relatively less explored. In this work, we present both on-policy and off-policy deterministic policy gradient theorems for the average reward performance criterion. Using these theorems, we also present an Average Reward Off-Policy Deep Deterministic Policy Gradient (ARO-DDPG) Algorithm. We first show asymptotic convergence analysis using the ODE-based method. Subsequently, we provide a finite time analysis of the resulting stochastic approximation scheme with linear function approximator and obtain an ε-optimal stationary policy with a sample complexity of Omega(ε-2.5). We compare the average reward performance of our proposed ARO-DDPG algorithm and observe better empirical performance compared to state-of-the-art on-policy average reward actor-critic algorithms over MuJoCo-based environments.