# Steering Language Models with Game-Theoretic Solvers

arXiv: https://arxiv.org/abs/2402.01704 · HTML: https://mlsanity.com/paper/2402.01704/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 50 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 29 of 29)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [AI safety via debate](https://mlsanity.com/paper/1805.00899/) (2018)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Do Prompt-Based Models Really Understand the Meaning of their Prompts?](https://mlsanity.com/paper/2109.01247/) (2021)
- [A Unified Game-Theoretic Approach to Multiagent Reinforcement Learning](https://mlsanity.com/paper/1711.00832/) (2017)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [AI Deception: A Survey of Examples, Risks, and Potential Solutions](https://mlsanity.com/paper/2308.14752/) (2023)
- [OpenSpiel: A Framework for Reinforcement Learning in Games](https://mlsanity.com/paper/1908.09453/) (2019)
- [Unsupervised Commonsense Question Answering with Self-Talk](https://mlsanity.com/paper/2004.05483/) (2020)
- [The Complexity of Constrained Min-Max Optimization](https://mlsanity.com/paper/2009.09623/) (2020)
- [Strategic Reasoning with Language Models](https://mlsanity.com/paper/2305.19165/) (2023)
- [Scalable AI Safety via Doubly-Efficient Debate](https://mlsanity.com/paper/2311.14125/) (2023)
- [The Consensus Game: Language Model Generation via Equilibrium Search](https://mlsanity.com/paper/2310.09139/) (2023)
- [The illusion of artificial inclusion](https://mlsanity.com/paper/2401.08572/) (2024)
- [Attention Flows are Shapley Value Explanations](https://mlsanity.com/paper/2105.14652/) (2021)
- [Modeling echo chambers and polarization dynamics in social networks](https://mlsanity.com/paper/1906.12325/) (2019)
- [Be Selfish, But Wisely: Investigating the Impact of Agent Personality in Mixed-Motive Human-Agent Interactions](https://mlsanity.com/paper/2310.14404/) (2023)
- [Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approaches](https://mlsanity.com/paper/2211.08371/) (2022)
- [Online Learning and Solving Infinite Games with an ERM Oracle](https://mlsanity.com/paper/2307.01689/) (2023)

## Built on by (most cited first, 28 of 28)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Emergent social conventions and collective bias in LLM populations](https://mlsanity.com/paper/2410.08948/) (2024)
- [LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models](https://mlsanity.com/paper/2404.01230/) (2024)
- [Game-theoretic LLM: Agent Workflow for Negotiation Games](https://mlsanity.com/paper/2411.05990/) (2024)
- [Will Systems of LLM Agents Cooperate: An Investigation into a Social Dilemma](https://mlsanity.com/paper/2501.16173/) (2025)
- [Game Theory Meets Large Language Models: A Systematic Survey with Taxonomy and New Frontiers](https://mlsanity.com/paper/2502.09053/) (2025)
- [Efficacy of Language Model Self-Play in Non-Zero-Sum Games](https://mlsanity.com/paper/2406.18872/) (2024)
- [Reevaluating Policy Gradient Methods for Imperfect-Information Games](https://mlsanity.com/paper/2502.08938/) (2025)
- [Policy Space Response Oracles: A Survey](https://mlsanity.com/paper/2403.02227/) (2024)
- [EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2502.12486/) (2025)
- [From Natural Language to Extensive-Form Game Representations](https://mlsanity.com/paper/2501.17282/) (2025)
- [Enhancing Language Model Rationality with Bi-Directional Deliberation Reasoning](https://mlsanity.com/paper/2407.06112/) (2024)
- [Multiplayer Federated Learning: Reaching Equilibrium with Less Communication](https://mlsanity.com/paper/2501.08263/) (2025)
- [Do LLMs Strategically Reveal, Conceal, and Infer Information? A Theoretical and Empirical Analysis in The Chameleon Game](https://mlsanity.com/paper/2501.19398/) (2025)
- [Distilling Game Code World Model Generation into Lightweight Large Language Models](https://mlsanity.com/paper/2605.24375/) (2026)
- [Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models](https://mlsanity.com/paper/2501.06248/) (2025)
- [The Power of Stories: Narrative Priming Shapes How LLM Agents Collaborate and Compete](https://mlsanity.com/paper/2505.03961/) (2025)
- [Code-Space Response Oracles: Generating Interpretable Multi-Agent Policies with Large Language Models](https://mlsanity.com/paper/2603.10098/) (2026)
- [Discovering Expert-Level Nash Equilibrium Algorithms with Large Language Models](https://mlsanity.com/paper/2508.11874/) (2025)
- [Make an Offer They Can't Refuse: Grounding Bayesian Persuasion in Real-World Dialogues without Pre-Commitment](https://mlsanity.com/paper/2510.13387/) (2025)
- [Game of Thought: Robust Information Seeking with Large Language Models Using Game Theory](https://mlsanity.com/paper/2602.01708/) (2026)
- [Safety Game: Inference-Time Alignment of Black-Box LLMs via Constrained Optimization](https://mlsanity.com/paper/2510.09330/) (2025)
- [Agentmandering: A Game-Theoretic Framework for Fair Redistricting via Large Language Model Agents](https://mlsanity.com/paper/2511.04076/) (2025)
- [LLM Active Alignment: A Nash Equilibrium Perspective](https://mlsanity.com/paper/2602.06836/) (2026)
- [Communicate-Predict-Act: Evaluating Social Intelligence of Agents](https://mlsanity.com/paper/2604.08727/) (2026)
- [How a Cooperative-Override Circuit Suppresses Nash Play in Large Language Models](https://mlsanity.com/paper/2604.27167/) (2026)
- [Strategy, Not Payoffs: A Behavioural Embedding of Normal-Form Games](https://mlsanity.com/paper/2607.27536/) (2026)
- [AI Alignment through a Game-theoretic Lens: A Survey](https://mlsanity.com/paper/2608.27910/) (2026)

## Abstract (arXiv)

Mathematical models of interactions among rational agents have long been studied in game theory. However these interactions are often over a small set of discrete game actions which is very different from how humans communicate in natural language. To bridge this gap, we introduce a framework that allows equilibrium solvers to work over the space of natural language dialogue generated by large language models (LLMs). Specifically, by modelling the players, strategies and payoffs in a "game" of dialogue, we create a binding from natural language interactions to the conventional symbolic logic of game theory. Given this binding, we can ask existing game-theoretic algorithms to provide us with strategic solutions (e.g., what string an LLM should generate to maximize payoff in the face of strategic partners or opponents), giving us predictors of stable, rational conversational strategies. We focus on three domains that require different negotiation strategies: scheduling meetings, trading fruit and debate, and evaluate an LLM's generated language when guided by solvers. We see that LLMs that follow game-theory solvers result in dialogue generations that are less exploitable than the control (no guidance from solvers), and the language generated results in higher rewards, in all negotiation domains. We discuss future implications of this work, and how game-theoretic solvers that can leverage the expressivity of natural language can open up a new avenue of guiding language research.