mlsanityData to 7 Oct 2026

Sample What You Say: Aligning Language Models to Sample the Distributions They State

arXiv 2609.34929 ↗First posted 28 Sep 2026cs.LG cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language models are increasingly used to sample from a specified distribution, for instance, to simulate survey respondents or generate synthetic data. Instruction-tuned models can state such a distribution correctly and still fail to sample from it. Prompting and changes to decoding reduce this mismatch only partly, which motivates training with policy optimization. Group relative policy optimization (GRPO) is a natural fit for this problem because it already samples a group of rollouts per prompt, and the group's empirical distribution can be compared with the target. However, scoring the group as a whole gives every rollout the same reward. Group-relative centering then sets all advantages to zero, and the model receives no learning signal. To give each rollout its own signal, we introduce the witness advantage, a per-rollout advantage derived from maximum mean discrepancy (MMD). It trains a model to match a target distribution over a finite set of outcomes. The MMD between the model's distribution and the target has a witness function that measures how over- or under-produced each outcome is. Each rollout's advantage estimates the negative witness at its outcome, so a rollout is rewarded for an outcome the group under-produces and penalized for one it over-produces. The witness advantage is computed in closed form from the group's outcome counts, and we use it as the reward in GRPO. On unseen target distributions, training with the witness advantage substantially reduces the total variation distance to the target while largely preserving the model's general capabilities.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 61 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

48 of 61 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Base Models Beat Aligned Models at Randomness and Creativity30 Apr
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 46

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2024Qwen2.5 Technical Reportcited by 4,713
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  10. 2023Whose Opinions Do Language Models Reflect?cited by 759
  11. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
34 more
  1. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  2. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  3. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 137
  4. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  5. 2020A Distributional Approach to Controlled Text Generationcited by 117
  6. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  7. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 100
  8. 2023Amortizing intractable inference in large language modelscited by 94
  9. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  10. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  11. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  12. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 78
  13. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  14. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 58
  15. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  16. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  17. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  18. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 36
  19. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  20. 2024Heads, Tails, and AI Fails: LLMs, Randomness, and Human Judgmentscited by 23
  21. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 19
  22. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  23. 2025Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerabilitycited by 17
  24. 2025String Seed of Thought: Prompting LLMs for Distribution-Faithful and Diverse Generationcited by 11
  25. 2025Group-Aware Reinforcement Learning for Output Diversity in Large Language Modelscited by 9
  26. 2025Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Samplingcited by 7
  27. 2026The Illusion of Stochasticity in LLMscited by 7
  28. 2026Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributionscited by 5
  29. 2026Beyond Mode Collapse: Distribution Matching for Diverse Reasoningcited by 5
  30. 2026Annotations Mitigate Post-Training Mode Collapsecited by 4
  31. 2026Probabilistic Calibration Is a Trainable Capability in Language Modelscited by 2
  32. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  33. 2026Alignment Defends LLMs from Property Inference Attackscited by 1
  34. 2026Instruction-Tuned Language Models Cannot Sample from Distributions They Can Describecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
  2. 2025Group Causal Policy Optimization for Post-Training Large Language Models
  3. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
  4. 2026Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL
  5. 2026Holder Policy Optimisation
  6. 2025Composing Policy Gradients and Prompt Optimization for Language Model Programs
  7. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  8. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning