# Cross-Entropy Games and Frost Training

arXiv: https://arxiv.org/abs/2605.27701 · HTML: https://mlsanity.com/paper/2605.27701/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 20 references that are in the map.
- First posted on arXiv 26 May 2026.

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [Amortizing intractable inference in large language models](https://mlsanity.com/paper/2310.04363/) (2023)
- [Gradient-Based Constrained Sampling from Language Models](https://mlsanity.com/paper/2205.12558/) (2022)
- [Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization](https://mlsanity.com/paper/2004.08763/) (2020)
- [RLP: Reinforcement as a Pretraining Objective](https://mlsanity.com/paper/2510.01265/) (2025)
- [CEM-GD: Cross-Entropy Method with Gradient Descent Planner for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2112.07746/) (2021)
- [TaylorGAN: Neighbor-Augmented Policy Update for Sample-Efficient Natural Language Generation](https://mlsanity.com/paper/2011.13527/) (2020)
- [Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures](https://mlsanity.com/paper/2506.06832/) (2025)
- [Cognitive Training for Language Models: Towards General Capabilities via Cross-Entropy Games](https://mlsanity.com/paper/2603.22479/) (2026)

## Abstract (arXiv)

We present Frost Training, a method for improving Monte Carlo-based policy optimization for a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The key idea is to exploit the gradient of the reward function in embedding space. This signal is used in the Greedy Coordinate Gradient (GCG) jailbreaking technique; we demonstrate for the first time that it can also be used to boost model training. We validate our method using GRPO training for maximum-likelihood infilling. Frost Training improves the model's ability to generate high-scoring outputs, reaching higher maximum scores in a best-of-k setting, and does so at an increased speed.