mlsanityData to 7 Oct 2026

ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation

arXiv 2603.21140 ↗First posted 22 Mar 2026cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

also in 1 more
Abstract (arXiv)

Training large language models (LLMs) with synthetic reasoning data has become a popular approach to enhancing their reasoning capabilities, while a key factor influencing the effectiveness of this paradigm is the quality of the generated multi-step reasoning data. To generate high-quality reasoning data, many recent methods generate synthetic reasoning paths and filter them based on final answer correctness, often overlooking flaws in intermediate reasoning steps. To enhance the verification of intermediate reasoning steps, prior work primarily resorts to code execution or symbolic reasoning engines. However, code-based validation is restricted to code or mathematical tasks, and reasoning engines require a well-structured and complete context. As a result, existing methods fail to function effectively in natural language reasoning tasks that involve ambiguous or incomplete contexts. In these tasks, synthetic data still lack reliable checks for verifying each reasoning step. To address this challenge, we introduce ORACLE, a structured data generation framework inspired by syllogistic reasoning. ORACLE integrates the generative strengths of LLMs with symbolic supervision: the LLM produces step-wise reasoning contexts, while a symbolic reasoning engine verifies the validity of each intermediate step. By employing a unified prompting template to elicit modular reasoning chains, ORACLE enables fine-grained, step-level validation, facilitating the construction of high-quality multi-step reasoning data. Across six logical, factual, and commonsense reasoning benchmarks, our ORACLE consistently outperforms strong baselines on multiple models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 9 of its 40 references that are in the map. First posted on arXiv 22 Mar 2026.

Builds on

9 of 40 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025SR-FoT: A Syllogistic-Reasoning Framework of Thought for Large Language Models Tackling Knowledge-based Reasoning Tasks20 Jan
  2. 2024DeepSeek-V3 Technical Report27 Dec
  3. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 6

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2023Qwen Technical Reportcited by 3,273
  3. 2023Mistral 7Bcited by 3,065
  4. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 417
  5. 2024LeanReasoner: Boosting Complex Logical Reasoning with Leancited by 28
  6. 2025Theorem Prover as a Judge for Synthetic Data Generationcited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpus
  2. 2024Enhancing Logical Reasoning in Large Language Models through Graph-based Synthetic Data
  3. 2026Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training
  4. 2025Can Large Language Models Learn Formal Logic? A Data-Driven Training and Evaluation Framework
  5. 2025Increasing LLM Coding Capabilities through Diverse Synthetic Coding Tasks
  6. 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation
  7. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  8. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling