mlsanityData to 7 Oct 2026

IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

arXiv 2607.01431 ↗First posted 1 Jul 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of reasoning-mode gains. Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving. Reasoning toggles on highly capable models provide less than 5 percentage points accuracy gain across all domains, and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2 percentage points) underperforms on ISOSCI (-24.7 percentage points), showing that benchmark choice determines conclusions about reasoning utility. We release ISOSCI at https://huggingface.co/datasets/isosci/isosci

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 30 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

23 of 30 references.

References · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  9. 2024FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AIcited by 227
  10. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  11. 2024Disentangling Memory and Reasoning Ability in Large Language Modelscited by 43
  12. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 28
11 more
  1. 2025A Tutorial on LLM Reasoning: Relevant Methods behind ChatGPT o1cited by 24
  2. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  3. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?cited by 21
  4. 2024SocialGPT: Prompting LLMs for Social Relation Reasoning via Greedy Segment Optimizationcited by 14
  5. 2025PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Levelcited by 14
  6. 2025Have Large Language Models Learned to Reason? A Characterization via 3-SAT Phase Transitioncited by 13
  7. 2025GPO: Learning from Critical Steps to Improve LLM Reasoningcited by 10
  8. 2025Theorem-of-Thought: A Multi-Agent Framework for Abductive, Deductive, and Inductive Reasoning in Language Modelscited by 6
  9. 2025QuantumBench: A Benchmark for Quantum Problem Solvingcited by 3
  10. 2026Reasoning Gets Harder for LLMs Inside A Dialoguecited by 2
  11. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
  2. 2026SciR: A Controllable Benchmark for Scientific Reasoning in LLMs
  3. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems
  4. 2026WildSci: Advancing Scientific Reasoning from In-the-Wild Literature
  5. 2026SciVQR: A Multidisciplinary Multimodal Benchmark for Advanced Scientific Reasoning Evaluation
  6. 2026ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure
  7. 2025RAISE: Enhancing Scientific Reasoning in LLMs via Step-by-Step Retrieval
  8. 2026PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs