# Actor-Critic or Critic-Actor? A Tale of Two Time Scales

arXiv: https://arxiv.org/abs/2210.04470 · HTML: https://mlsanity.com/paper/2210.04470/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 27 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Built on by (most cited first, 7 of 7)

- [Two-Timescale Critic-Actor for Average Reward MDPs with Function Approximation](https://mlsanity.com/paper/2402.01371/) (2024)
- [The Actor-Critic Update Order Matters for PPO in Federated Reinforcement Learning](https://mlsanity.com/paper/2506.01261/) (2025)
- [Reinforcement Learning with Quasi-Hyperbolic Discounting](https://mlsanity.com/paper/2409.10583/) (2024)
- [Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem](https://mlsanity.com/paper/2508.13963/) (2025)
- [Finite Time Analysis of Constrained Natural Critic-Actor Algorithm with Improved Sample Complexity](https://mlsanity.com/paper/2510.04189/) (2025)
- [Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum](https://mlsanity.com/paper/2602.01505/) (2026)
- [A Concentration Bound for Two-Timescale Actor-Critic Algorithm](https://mlsanity.com/paper/2609.29117/) (2026)

## Abstract (arXiv)

We revisit the standard formulation of tabular actor-critic algorithm as a two time-scale stochastic approximation with value function computed on a faster time-scale and policy computed on a slower time-scale. This emulates policy iteration. We observe that reversal of the time scales will in fact emulate value iteration and is a legitimate algorithm. We provide a proof of convergence and compare the two empirically with and without function approximation (with both linear and nonlinear function approximators) and observe that our proposed critic-actor algorithm performs on par with actor-critic in terms of both accuracy and computational effort.