mlsanityData to 13 Oct 2026

Hierarchical Deep Counterfactual Regret Minimization

arXiv 2305.17327 ↗First posted 27 May 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Imperfect Information Games (IIGs) are used to model games under uncertainty or lack complete information. Counterfactual Regret Minimization (CFR) is one of the most successful families of algorithms for IIGs. The integration of skill-based strategy learning with CFR could potentially mirror more human-like decision-making and improve learning on complex IIGs. It enables the learning of a hierarchical strategy, wherein low-level components represent skills for solving subgames and the high-level component manages the transition between skills. In this paper, we introduce the first hierarchical version of Deep CFR (HDCFR), an innovative method that boosts learning efficiency in tasks involving extensively large state spaces and deep game trees. Notably, HDCFR enables learning with predefined (human) expertise and extracting skills transferable to similar tasks. We first present the algorithm and establish its theory in a tabular setting, including hierarchical CFR update rules and a variance-reduced Monte Carlo sampling extension for the model-free setting, where backtracking is infeasible. We then extend HDCFR to large-scale tasks via deep learning objectives that match the tabular targets under exact function fitting. Code: https://anonymous.4open.science/r/HDCFR_RUN-677B.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 22 of its 44 references that are in the map. First posted on arXiv 27 May 2023.

Builds on

22 of 44 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 739
  3. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 518
  4. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  5. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  6. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 155
  7. 2018Deep Counterfactual Regret Minimizationcited by 102
  8. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  9. 2017Safe and Nested Subgame Solving for Imperfect-Information Gamescited by 39
  10. 2020DREAM: Deep Regret minimization with Advantage baselines and Model-free learningcited by 36
  11. 2018Actor-Critic Policy Optimization in Partially Observable Multiagent Environmentscited by 35
  12. 2018Depth-Limited Solving for Imperfect-Information Gamescited by 24
10 more
  1. 2018Variance Reduction in Monte Carlo Counterfactual Regret Minimization (VR-MCCFR) for Extensive Form Games using Baselinescited by 24
  2. 2019Single Deep Counterfactual Regret Minimizationcited by 20
  3. 2021Adversarial Option-Aware Hierarchical Imitation Learningcited by 19
  4. 2018Double Neural Counterfactual Regret Minimizationcited by 16
  5. 2022Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Controlcited by 16
  6. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15
  7. 2014Solving Games with Functional Regret Estimationcited by 12
  8. 2019Low-Variance and Zero-Variance Baselines for Extensive-Form Gamescited by 9
  9. 2022Learning Multi-agent Skills for Tabular Reinforcement Learning using Factor Graphscited by 4
  10. 2021D2CFR: Minimize Counterfactual Regret with Deep Dueling Neural Networkcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  2. 2024Variational Offline Multi-agent Skill Discoverycited by 6
  3. 2025MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planningcited by 5
  4. 2025Policy-Driven World Model Adaptation for Robust Offline Model-based Reinforcement Learningcited by 2