# PokerBench: Training Large Language Models to become Professional Poker Players

arXiv: https://arxiv.org/abs/2501.08328 · HTML: https://mlsanity.com/paper/2501.08328/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jan 2025.

## Built on by (most cited first, 18 of 18)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models](https://mlsanity.com/paper/2506.01341/) (2025)
- [AIM-Bench: Evaluating Decision-making Biases of Agentic LLM as Inventory Manager](https://mlsanity.com/paper/2508.11416/) (2025)
- [PokerSkill: LLMs Can Play Expert-Level Poker without Training or Solvers](https://mlsanity.com/paper/2605.30094/) (2026)
- [GTO Wizard Benchmark](https://mlsanity.com/paper/2603.23660/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Solver-Guided Reasoning for Mixed-Equilibrium Strategies](https://mlsanity.com/paper/2608.06741/) (2026)
- [Game Arena: Strategic LLM Evaluation in Competitive Environments](https://mlsanity.com/paper/2609.31473/) (2026)
- [Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review](https://mlsanity.com/paper/2507.10142/) (2025)
- [SpinGPT: A Large-Language-Model Approach to Playing Poker Correctly](https://mlsanity.com/paper/2509.22387/) (2025)
- [How Clued up are LLMs? Evaluating Multi-Step Deductive Reasoning in a Text-Based Game Environment](https://mlsanity.com/paper/2603.17169/) (2026)
- [LUDOBENCH: Evaluating LLM Behavioural Decision-Making Through Spot-Based Board Game Scenarios in Ludo](https://mlsanity.com/paper/2604.05681/) (2026)
- [AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games](https://mlsanity.com/paper/2608.06362/) (2026)
- [Abstraction Agent](https://mlsanity.com/paper/2609.04303/) (2026)
- [Clueing up LLMs with Tool-Augmented Deductive Reasoning](https://mlsanity.com/paper/2609.18736/) (2026)
- [RSD-Poker: Structure-Adaptive and Shift-Robust Risk-Utility Certification for Residual Policies in Imperfect-Information Games](https://mlsanity.com/paper/2609.33669/) (2026)

## Abstract (arXiv)

We introduce PokerBench - a benchmark for evaluating the poker-playing abilities of large language models (LLMs). As LLMs excel in traditional NLP tasks, their application to complex, strategic games like poker poses a new challenge. Poker, an incomplete information game, demands a multitude of skills such as mathematics, reasoning, planning, strategy, and a deep understanding of game theory and human psychology. This makes Poker the ideal next frontier for large language models. PokerBench consists of a comprehensive compilation of 11,000 most important scenarios, split between pre-flop and post-flop play, developed in collaboration with trained poker players. We evaluate prominent models including GPT-4, ChatGPT 3.5, and various Llama and Gemma series models, finding that all state-of-the-art LLMs underperform in playing optimal poker. However, after fine-tuning, these models show marked improvements. We validate PokerBench by having models with different scores compete with each other, demonstrating that higher scores on PokerBench lead to higher win rates in actual poker games. Through gameplay between our fine-tuned model and GPT-4, we also identify limitations of simple supervised fine-tuning for learning optimal playing strategy, suggesting the need for more advanced methodologies for effectively training language models to excel in games. PokerBench thus presents a unique benchmark for a quick and reliable evaluation of the poker-playing ability of LLMs as well as a comprehensive benchmark to study the progress of LLMs in complex game-playing scenarios.