# IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

arXiv: https://arxiv.org/abs/2607.01431 · HTML: https://mlsanity.com/paper/2607.01431/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 1 Jul 2026.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 23 of 23)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI](https://mlsanity.com/paper/2411.04872/) (2024)
- [Guiding Language Model Reasoning with Planning Tokens](https://mlsanity.com/paper/2310.05707/) (2023)
- [Disentangling Memory and Reasoning Ability in Large Language Models](https://mlsanity.com/paper/2411.13504/) (2024)
- [Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoning](https://mlsanity.com/paper/2502.07154/) (2025)
- [A Tutorial on LLM Reasoning: Relevant Methods behind ChatGPT o1](https://mlsanity.com/paper/2502.10867/) (2025)
- [From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?](https://mlsanity.com/paper/2506.08295/) (2025)
- [The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2503.23084/) (2025)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimization](https://mlsanity.com/paper/2410.21411/) (2024)
- [Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transition](https://mlsanity.com/paper/2504.03930/) (2025)
- [GPO: Learning from Critical Steps to Improve LLM Reasoning](https://mlsanity.com/paper/2509.16456/) (2025)
- [Theorem-of-Thought: A Multi-Agent Framework for Abductive, Deductive, and Inductive Reasoning in Language Models](https://mlsanity.com/paper/2506.07106/) (2025)
- [QuantumBench: A Benchmark for Quantum Problem Solving](https://mlsanity.com/paper/2511.00092/) (2025)
- [Reasoning Gets Harder for LLMs Inside A Dialogue](https://mlsanity.com/paper/2603.20133/) (2026)
- [Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks](https://mlsanity.com/paper/2604.12379/) (2026)

## Abstract (arXiv)

We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of reasoning-mode gains. Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving. Reasoning toggles on highly capable models provide less than 5 percentage points accuracy gain across all domains, and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2 percentage points) underperforms on ISOSCI (-24.7 percentage points), showing that benchmark choice determines conclusions about reasoning utility. We release ISOSCI at https://huggingface.co/datasets/isosci/isosci