# ARCHE: A Novel Task to Evaluate LLMs on Latent Reasoning Chain Extraction

arXiv: https://arxiv.org/abs/2511.12485 · HTML: https://mlsanity.com/paper/2511.12485/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 40 references that are in the map.
- First posted on arXiv 16 Nov 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 20 of 20)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Accelerating scientific discovery with Co-Scientist](https://mlsanity.com/paper/2502.18864/) (2025)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [AutoSurvey: Large Language Models Can Automatically Write Surveys](https://mlsanity.com/paper/2406.10252/) (2024)
- [DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process](https://mlsanity.com/paper/2503.08569/) (2025)
- [CRISPR-GPT for Agentic Automation of Gene-editing Experiments](https://mlsanity.com/paper/2404.18021/) (2024)
- [Intern-S1: A Scientific Multimodal Foundation Model](https://mlsanity.com/paper/2508.15763/) (2025)
- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [Logical Reasoning over Natural Language as Knowledge Representation: A Survey](https://mlsanity.com/paper/2303.12023/) (2023)
- [DeepResearch Arena: The First Exam of LLMs' Research Abilities via Seminar-Grounded Tasks](https://mlsanity.com/paper/2509.01396/) (2025)
- [Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Models](https://mlsanity.com/paper/2411.02382/) (2024)
- [A Survey on Hypothesis Generation for Scientific Discovery in the Era of Large Language Models](https://mlsanity.com/paper/2504.05496/) (2025)
- [GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation](https://mlsanity.com/paper/2503.12600/) (2025)
- [PhysUniBench: A Multi-Modal Physics Reasoning Benchmark at Undergraduate Level](https://mlsanity.com/paper/2506.17667/) (2025)
- [Can Large Language Models Help Experimental Design for Causal Discovery?](https://mlsanity.com/paper/2503.01139/) (2025)
- [SciReasoner: Laying the Scientific Reasoning Ground Across Disciplines](https://mlsanity.com/paper/2509.21320/) (2025)

## Built on by (most cited first, 3 of 3)

- [Argument Reconstruction as Supervision for Critical Thinking in LLMs](https://mlsanity.com/paper/2603.17432/) (2026)
- [LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation](https://mlsanity.com/paper/2605.25964/) (2026)
- [PEARL: Auditable Repair for Scientific Reasoning Graph Extraction](https://mlsanity.com/paper/2607.17917/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used in scientific domains. While they can produce reasoning-like content via methods such as chain-of-thought prompting, these outputs are typically unstructured and informal, obscuring whether models truly understand the fundamental reasoning paradigms that underpin scientific inference. To address this, we introduce a novel task named Latent Reasoning Chain Extraction (ARCHE), in which models must decompose complex reasoning arguments into combinations of standard reasoning paradigms in the form of a Reasoning Logic Tree (RLT). In RLT, all reasoning steps are explicitly categorized as one of three variants of Peirce's fundamental inference modes: deduction, induction, or abduction. To facilitate this task, we release ARCHE Bench, a new benchmark derived from 70 Nature Communications articles, including more than 1,900 references and 38,000 viewpoints. We propose two logic-aware evaluation metrics: Entity Coverage (EC) for content completeness and Reasoning Edge Accuracy (REA) for step-by-step logical validity. Evaluations on 10 leading LLMs on ARCHE Bench reveal that models exhibit a trade-off between REA and EC, and none are yet able to extract a complete and standard reasoning chain. These findings highlight a substantial gap between the abilities of current reasoning models and the rigor required for scientific argumentation.