# RL-CFR: Improving Action Abstraction for Imperfect Information Extensive-Form Games with Reinforcement Learning

arXiv: https://arxiv.org/abs/2403.04344 · HTML: https://mlsanity.com/paper/2403.04344/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 72 references that are in the map.
- First posted on arXiv 7 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 21 of 21)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Combining Deep Reinforcement Learning and Search for Imperfect-Information Games](https://mlsanity.com/paper/2007.13544/) (2020)
- [Deep Counterfactual Regret Minimization](https://mlsanity.com/paper/1811.00164/) (2018)
- [Solving Large Imperfect Information Games Using CFR+](https://mlsanity.com/paper/1407.5042/) (2014)
- [Solving Imperfect-Information Games via Discounted Regret Minimization](https://mlsanity.com/paper/1809.04040/) (2018)
- [From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularization](https://mlsanity.com/paper/2002.08456/) (2020)
- [Safe and Nested Subgame Solving for Imperfect-Information Games](https://mlsanity.com/paper/1705.02955/) (2017)
- [Solving Imperfect Information Games Using Decomposition](https://mlsanity.com/paper/1303.4441/) (2013)
- [Depth-Limited Solving for Imperfect-Information Games](https://mlsanity.com/paper/1805.08195/) (2018)
- [The Power of Regularization in Solving Extensive-Form Games](https://mlsanity.com/paper/2206.09495/) (2022)
- [Measuring the Size of Large No-Limit Poker Games](https://mlsanity.com/paper/1302.7008/) (2013)
- [Solving Games with Functional Regret Estimation](https://mlsanity.com/paper/1411.7974/) (2014)
- [Modular Architecture for StarCraft II with Deep Reinforcement Learning](https://mlsanity.com/paper/1811.03555/) (2018)
- [AIVAT: A New Variance Reduction Technique for Agent Evaluation in Imperfect Information Games](https://mlsanity.com/paper/1612.06915/) (2016)
- [Unlocking the Potential of Deep Counterfactual Value Networks](https://mlsanity.com/paper/2007.10442/) (2020)
- [Monte Carlo Continual Resolving for Online Strategy Computation in Imperfect Information Games](https://mlsanity.com/paper/1812.07351/) (2018)
- [Alternative Function Approximation Parameterizations for Solving Games: An Analysis of f-Regression Counterfactual Regret Minimization](https://mlsanity.com/paper/1912.02967/) (2019)
- [The Category Gm for Extensive-Form Games](https://mlsanity.com/paper/2105.11398/) (2021)
- [CFR-p: Counterfactual Regret Minimization with Hierarchical Policy Abstraction, and its Application to Two-player Mahjong](https://mlsanity.com/paper/2307.12087/) (2023)

## Built on by (most cited first, 10 of 10)

- [Real-Time Parallel Counterfactual Regret Minimization](https://mlsanity.com/paper/2605.19928/) (2026)
- [PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers](https://mlsanity.com/paper/2605.30094/) (2026)
- [Agents That Certify Their Own Exploits: Confidence-Scheduled Restricted Responses for Safe Opponent Exploitation](https://mlsanity.com/paper/2607.28520/) (2026)
- [Look-ahead Reasoning with a Learned Model in Imperfect Information Games](https://mlsanity.com/paper/2510.05048/) (2025)
- [ICM Out! Better Tournament Strategy from Computed Continuations, vs. Solvers and LLMs](https://mlsanity.com/paper/2608.09586/) (2026)
- [AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games](https://mlsanity.com/paper/2608.06362/) (2026)
- [Abstraction Agent](https://mlsanity.com/paper/2609.04303/) (2026)
- [No-Regret Strategy Solving in Imperfect-Information Games via Pre-Trained Embedding](https://mlsanity.com/paper/2511.12083/) (2025)
- [Self-Play Reinforcement Learning under Imperfect Information in Big 2](https://mlsanity.com/paper/2605.28863/) (2026)
- [GPU-CFR: 80x Faster Counterfactual Regret Minimization by Compiling the Game to Static Dataflow and CUDA Graph Replay](https://mlsanity.com/paper/2609.11923/) (2026)

## Abstract (arXiv)

Effective action abstraction is crucial in tackling challenges associated with large action spaces in Imperfect Information Extensive-Form Games (IIEFGs). However, due to the vast state space and computational complexity in IIEFGs, existing methods often rely on fixed abstractions, resulting in sub-optimal performance. In response, we introduce RL-CFR, a novel reinforcement learning (RL) approach for dynamic action abstraction. RL-CFR builds upon our innovative Markov Decision Process (MDP) formulation, with states corresponding to public information and actions represented as feature vectors indicating specific action abstractions. The reward is defined as the expected payoff difference between the selected and default action abstractions. RL-CFR constructs a game tree with RL-guided action abstractions and utilizes counterfactual regret minimization (CFR) for strategy derivation. Impressively, it can be trained from scratch, achieving higher expected payoff without increased CFR solving time. In experiments on Heads-up No-limit Texas Hold'em, RL-CFR outperforms ReBeL's replication and Slumbot, demonstrating significant win-rate margins of 64± 11 and 84± 17 mbb/hand, respectively.