Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Models
Not on the map. Why?
Abstract (arXiv)
Large language models must satisfy hard orthographic constraints during controlled text generation, yet systematic cross-family evaluation remains limited. We evaluate 39 configurations spanning three model families (Qwen3, Claude Haiku 4.5, GPT-5-mini) on 58 word puzzles requiring character-level constraint satisfaction. Cross-family differences produce substantially larger performance gaps (2.0-2.2x, F1 = 0.761 vs. 0.343) than parameter scaling within families (83% gain from 4B to 32B scaling), and a partial-correlation analysis rules out tokenizer design as a confound for within-family scaling. Thinking budget sensitivity proves heterogeneous: high-capacity models show strong returns (+0.102 to +0.136 F1), while mid-sized variants saturate or degrade, showing inconsistent compute benefits. Using difficulty ratings from 10,000 human solvers per puzzle, we establish modest but consistent calibration (rho = 0.28-0.42) across all families, yet identify systematic failures on common words with unusual orthography ("data", "loll", "acai": 83-91% human success, 94-98% model miss rate). These failures point to over-reliance on distributional plausibility that penalizes orthographically atypical but constraint-valid patterns.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 26 Nov 2025.
Builds on
14 of 30 references.
References · 14
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
- 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
- 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
- 2022Interpreting Language Models with Contrastive Explanationscited by 91
- 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
- 2025Flexible and Efficient Grammar-Constrained Decodingcited by 42
- 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 28
- 2022Why is constrained neural language generation particularly challenging?cited by 19
- 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
- 2024Language Models are Crossword Solverscited by 5
- 2024Beyond Film Subtitles: Is YouTube the Best Approximation of Spoken Vocabulary?cited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
- 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
- 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
- 2023Large Language Model Alignment: A Survey
- 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors
- 2024From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification
- 2025Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?
- 2025Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments