# LogicSkills: A Structured Benchmark for Formal Reasoning in Large Language Models

arXiv: https://arxiv.org/abs/2602.06533 · HTML: https://mlsanity.com/paper/2602.06533/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 2 of its 23 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 2 of 2)

- [Logical Reasoning over Natural Language as Knowledge Representation: A Survey](https://mlsanity.com/paper/2303.12023/) (2023)
- [Propositional Interpretability in Artificial Intelligence](https://mlsanity.com/paper/2501.15740/) (2025)

## Built on by (most cited first, 4 of 4)

- [ChLogic: Evaluating Robustness of Logical Reasoning in Chinese Expressions](https://mlsanity.com/paper/2606.17905/) (2026)
- [Diagnosis-Driven Automatic Repair for Agentic Workflow via Symbolic Inference](https://mlsanity.com/paper/2607.02882/) (2026)
- [Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoning](https://mlsanity.com/paper/2608.05139/) (2026)
- [ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning](https://mlsanity.com/paper/2609.38409/) (2026)

## Abstract (arXiv)

Large language models perform well on many logical reasoning benchmarks, but it remains unclear which core logical skills they truly master. To address this, we introduce LogicSkills, a benchmark that isolates three fundamental logical skills: (i) textitformal symbolizationunicodex2014translating premises into first-order logic; (ii) textitcountermodel constructionunicodex2014showing that an argument is logically invalid by constructing a finite countermodel; and (iii) textitvalidity assessmentunicodex2014determining whether a conclusion follows from a set of premises. Items are drawn from the two-variable fragment of first-order logic without identity and are presented in both English and a Carrollian nonce-word language. All instances are solver-verified with Z3 for correctness and non-triviality. Across conventional instruction-tuned LLMs, performance is high on textitvalidity assessment but substantially lower on textitformal symbolization and textitcountermodel construction, highlighting that high task-level accuracy can mask weaknesses in core logical skills. In contrast, recent reasoning-tuned models perform strongly across all three tasks, suggesting a more systematic logical skill profile.