A Gold-Standard Study of What Makes a Lightweight Game-Playing Agent Strong

arXiv 2607.06854 ↗First posted 7 Jul 2026cs.LG cs.AI cs.GTcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement learning agents for imperfect-information card games are only as strong as the opponents they train against, and they are hard to grade, since they beat a random opponent over 99 percent of the time and only tie copies of themselves. So we build a strong, fixed, rule-based expert for Gin Rummy and use it only as a yardstick, never for training. It beats every agent we trained 70 to 99 percent of the time. Across more than a hundred runs, we isolate what makes a lightweight agent stronger. Trust region updates, a well-aimed reward, a curriculum of tougher opponents, warm starting, and keeping the best checkpoint all help, and stacking them lifts a self-play champion from about 30 to 36 percent against the expert. Several ideas did not pay off. Short-term and longer-term reward shaping, learned state embeddings, imitation and DAgger, and a live large language model opponent were each unhelpful, too slow, or too heavy to train at scale. Comparing MLP, convolutional, set-based, attention, and recurrent encoders shows that extra capacity does little to break the ceiling, suggesting the limit is information rather than network size. We add standard baselines (neural fictitious self-play and information set Monte Carlo search) and confirm the approach carries over to Leduc Hold'em, where the optimum is computable. The result is a lightweight, game-agnostic recipe that trains competitive agents without training on the expert, for any game a small model can handle, reported with robust statistics and released as a reusable package.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 37 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

20 of 37 references.

References · 20

2017Attention Is All You Needcited by 60,361
2017Proximal Policy Optimization Algorithmscited by 15,215
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
2015Trust Region Policy Optimizationcited by 2,580
2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
2017Deep Setscited by 1,421
2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2017Deep Reinforcement Learning that Matterscited by 754
2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
2019Causal Confusion in Imitation Learningcited by 288
2024Reinforcement learningcited by 247

8 more

2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 220
2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 158
2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 114
2018Deep Counterfactual Regret Minimizationcited by 103
2021DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learningcited by 66
2020Suphx: Mastering Mahjong with Deep Reinforcement Learningcited by 58

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way