# Constraint-Aware Training

arXiv: https://arxiv.org/abs/2610.02909 · HTML: https://mlsanity.com/paper/2610.02909/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 10 references that are in the map.
- First posted on arXiv 2 Oct 2026.

## Builds on (most cited first, 8 of 8)

- [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](https://mlsanity.com/paper/2305.13971/) (2023)
- [A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers](https://mlsanity.com/paper/2503.03961/) (2025)
- [Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages](https://mlsanity.com/paper/2310.13897/) (2023)
- [Type-Constrained Code Generation with Language Models](https://mlsanity.com/paper/2504.09246/) (2025)
- [ChopChop: a Programmable Framework for Semantically Constraining the Output of Language Models](https://mlsanity.com/paper/2509.00360/) (2025)
- [Grammar-Forced Translation of Natural Language to Temporal Logic using LLMs](https://mlsanity.com/paper/2512.16814/) (2025)
- [Eliminating Out-of-Domain Recommendations in LLM-based Recommender Systems: A Unified View](https://mlsanity.com/paper/2505.03336/) (2025)
- [Context-Free Recognition with Transformers](https://mlsanity.com/paper/2601.01754/) (2026)

## Abstract (arXiv)

When generating programs with language models, constrained decoding can apply program analyses to exclude tokens that violate syntax, scope, or typing rules. However, there is a duplication: standard training already teaches the model to suppress the tokens rejected by these analyses. This duplication leads to the question: if we will perform some analysis to filter a set tokens out during inference anyways, can we avoid teaching the model the said analysis altogether during training, and does this externalization lead to more efficient models? This paper defines a general constraint-aware objective satisfying this externalization desideratum and formalizes the benefits of externalization into three concrete theorems about model size and data efficiency. We show, through a controlled synthetic experiment, that the theorems survive training dynamics: constraint-aware training yields lower prediction loss at a matched parameter count and data compared to ordinary cross-entropy training, motivating training objectives that incorporate the analyses used during generation.