Solver-Hard Is Not Model-Hard: A Hardness-Controlled Diagnostic for LLM Constraint Reasoning
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
LLM constraint reasoners are often evaluated near the random-SAT phase transition, confounding density and solver hardness. We test instance-level transfer while near-matching clause density. At aligned size bins, with near-matched density and matched maximum clause width, we compare proof-hard expander-Tseitin and proof-easy ladder-Tseitin formulas, pigeonhole anchors, and density-mismatched controls. Theory separates their resolution hardness; a solver-specific Glucose mean-conflict proxy differs by up to 51×, and five other solvers preserve the direction. Across three included models (243 instances each; a fourth is excluded for abstention), the near-matched-density accuracy gaps range from -32 to +20 points, with a pooled gap of +1.7 points (p=0.74) and a wrong-signed correctness-versus-conflict association (r=+0.15). A proof-preserving relabeling lowers accuracy in all five clusters for one model (mean -93 points) but not another, exposing model-surface sensitivity. In a preregistered extension, provider-reported completion-token spend does not consistently increase with the proxy after accounting for formula length and censoring. At 16k, the reasoning model spends more on proof-easy matched formulas and exhausts its budget on the solver-easiest UNSAT family; the 32k C1 gap is absent. These scoped dissociations concern verdict accuracy and observed token spend, not certificate solving, exact proof length, or allocation efficiency.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 21 of its 28 references that are in the map. First posted on arXiv 19 Jul 2026.
Builds on
21 of 28 references.
Closest ancestors · 8
Same direction, newest first
- 2026Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability27 May
- 2026Evaluating Robustness of Reasoning Models on Parameterized Logical Problems13 Feb
- 2025SATQuest: A Verifier for Logical Reasoning Evaluation and Reinforcement Fine-Tuning of LLMs31 Aug
- 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning14 Jul
- 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulas20 May
- 2025Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transition4 Apr
- 2024Can Transformers Reason Logically? A Study in SAT Solving9 Oct
- 2024Can Large Language Models Reason? A Characterization via 3-SAT13 Aug
Other references · 13
- 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
- 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
- 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 298
- 2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52
- 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
- 2025Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!cited by 36
- 2025THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating Overthinking in Reasoning Modelscited by 34
- 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 30
- 2023Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspectivecited by 26
- 2026When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scalingcited by 13
- 2026Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimizationcited by 8
- 2025Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfactioncited by 6
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs
- 2025A Reality Check of Language Models as Formalizers on Constraint Satisfaction Problems
- 2026Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism
- 2025FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
- 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning
- 2026ConstraintBench: Benchmarking LLM Constraint Reasoning on Direct Optimization
- 2026Understanding CDCL Solvers via Scalability Studies and Proofdoors ↗
- 2025Hard2Verify: A Step-Level Verification Benchmark for Open-Ended Frontier Math