# Distributional Alignment Games for Answer-Level Fine-Tuning

arXiv: https://arxiv.org/abs/2604.27166 · HTML: https://mlsanity.com/paper/2604.27166/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 8 of its 15 references that are in the map.
- First posted on arXiv 29 Apr 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 8 of 8)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Solving math word problems with process- and outcome-based feedback](https://mlsanity.com/paper/2211.14275/) (2022)
- [Jointly Reinforcing Diversity and Quality in Language Model Generations](https://mlsanity.com/paper/2509.02534/) (2025)
- [Coherence Mechanisms for Provable Self-Improvement](https://mlsanity.com/paper/2511.08440/) (2025)

## Built on by (most cited first, 3 of 3)

- [Generalized Distributional Alignment Games for Unbiased Answer-Level Fine-Tuning](https://mlsanity.com/paper/2605.02435/) (2026)
- [MaD-RL: Matching Distributions for Calibrating LLMs with Reinforcement Learning](https://mlsanity.com/paper/2609.31644/) (2026)
- [Train for Many, Update with One: Improving Multi-Sample Reasoning via Fenchel-Bregman Posterior Learning](https://mlsanity.com/paper/2610.17735/) (2026)

## Abstract (arXiv)

We focus on the problem of Answer-Level Fine-Tuning (ALFT), where the goal is to optimize a language model based on the correctness or properties of its final answers, rather than the specific reasoning traces used to produce them. Directly optimizing answer-level objectives is computationally intractable due to the need to marginalize over the vast space of latent reasoning paths. To overcome this, we propose a general game-theoretical framework that lifts the problem to a Distributional Alignment Game. We formulate ALFT as a two-player game between a Policy (the generator) and a Target (an auxiliary distribution). We prove that the Nash Equilibrium of this game corresponds exactly to the solution of the original answer-level optimization problem. This variational perspective transforms the intractable marginalization problem into a tractable projection problem. We demonstrate that this framework unifies recent approaches to diversity and self-improvement (coherence) and provide efficient algorithms compatible with Group Relative Policy Optimization (GRPO), such as Coherence-GRPO, yielding significant complexity gains in mathematical reasoning tasks.