mlsanityData to 14 Oct 2026

Toward Honest Language Models for Deductive Reasoning

arXiv 2511.09222 ↗First posted 12 Nov 2025cs.CLcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Deductive reasoning is the process of deriving conclusions strictly from the given premises, without relying on external knowledge. We define honesty in this setting as a model's ability to respond only when the conclusion is logically entailed by the premises, and to abstain otherwise. However, current language models often fail to reason honestly, producing unwarranted answers when the input is insufficient. To study this challenge, we formulate honest deductive reasoning as multi-step tasks where models must either derive the correct conclusion or abstain. We curate two datasets from graph structures, one for linear algebra and one for logical inference, and introduce unanswerable cases by randomly perturbing an edge in half of the instances. We find that prompting and existing training methods, including GRPO with or without supervised fine-tuning initialization, struggle on these tasks. In particular, GRPO optimize only for final task outcomes, leaving models vulnerable to collapse when negative rewards dominate early training. To address this, we propose ACNCHOR, a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse. Our results demonstrate that this method stabilizes learning and significantly improves the overall reasoning performance, underscoring the importance of training dynamics for enabling honest deductive reasoning in language models.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 32 of its 45 references that are in the map. First posted on arXiv 12 Nov 2025.

Builds on

32 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions10 Jun
  2. 2024Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models24 Jun

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  11. 2024Alignment faking in large language modelscited by 360
  12. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
18 more
  1. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  2. 2025Learning to Reason under Off-Policy Guidancecited by 281
  3. 2023Do Large Language Models Know What They Don't Know?cited by 243
  4. 2025Why Language Models Hallucinatecited by 220
  5. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  6. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  7. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  8. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  9. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 75
  10. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Samplingcited by 60
  11. 2025GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learningcited by 34
  12. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoningcited by 30
  13. 2025TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoningcited by 28
  14. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  15. 2025Why Do Some Language Models Fake Alignment While Others Don't?cited by 20
  16. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Modelscited by 18
  17. 2025Proximal Supervised Fine-Tuningcited by 16
  18. 2025TreeCut: A Synthetic Unanswerable Math Word Problem Dataset for LLM Hallucination Evaluationcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injectioncited by 3
  2. 2026Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learningcited by 1
  3. 2026IdeaScientist: Orchestrating Agents for Grounded Scientific Ideationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Reason Efficiently with A* Post-Training
  2. 2022FaiRR: Faithful and Robust Deductive Reasoning over Natural Language
  3. 2022Can Pretrained Language Models (Yet) Reason Deductively?
  4. 2025Deductive Logic in Language Models: Horizontal vs Vertical Reasoning
  5. 2023Examining the Emergence of Deductive Reasoning in Generative Language Models
  6. 2026DoubtLess: Training-Free Contrastive Decoding for Efficient Reasoning
  7. 2025Logical Reasoning with Outcome Reward Models for Test-Time Scaling
  8. 2025Concise Reasoning via Reinforcement Learning