# Variance Adjusted Actor Critic Algorithms

arXiv: https://arxiv.org/abs/1310.3697 · HTML: https://mlsanity.com/paper/1310.3697/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2013.

## Built on by (most cited first, 11 of 11)

- [State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards](https://mlsanity.com/paper/2102.11941/) (2021)
- [Challenging Common Assumptions in Convex Reinforcement Learning](https://mlsanity.com/paper/2202.01511/) (2022)
- [Variance Penalized On-Policy and Off-Policy Actor-Critic](https://mlsanity.com/paper/2102.01985/) (2021)
- [Towards Multi-Agent Reinforcement Learning driven Over-The-Counter Market Simulations](https://mlsanity.com/paper/2210.07184/) (2022)
- [High-Confidence Off-Policy (or Counterfactual) Variance Estimation](https://mlsanity.com/paper/2101.09847/) (2021)
- [Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problem](https://mlsanity.com/paper/2410.05673/) (2024)
- [ACReL: Adversarial Conditional value-at-risk Reinforcement Learning](https://mlsanity.com/paper/2109.09470/) (2021)
- [Markov Chain Variance Estimation: A Stochastic Approximation Approach](https://mlsanity.com/paper/2409.05733/) (2024)
- [A Unified Off-Policy Evaluation Approach for General Value Function](https://mlsanity.com/paper/2107.02711/) (2021)
- [A Subgame Perfect Equilibrium Reinforcement Learning Approach to Time-inconsistent Problems](https://mlsanity.com/paper/2110.14295/) (2021)
- [Nonparametric Variance-Penalized Actor-Critic: Statistical Inference for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2609.14327/) (2026)

## Abstract (arXiv)

We present an actor-critic framework for MDPs where the objective is the variance-adjusted expected return. Our critic uses linear function approximation, and we extend the concept of compatible features to the variance-adjusted setting. We present an episodic actor-critic algorithm and show that it converges almost surely to a locally optimal point of the objective function.