mlsanityData to 7 Oct 2026

Conceptual and Unbiased Reasoning in Language Models

arXiv 2404.00205 ↗First posted 30 Mar 2024cs.CLcited by 19

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Conceptual reasoning, the ability to reason in abstract and high-level perspectives, is key to generalization in human cognition. However, limited study has been done on large language models' capability to perform conceptual reasoning. In this work, we bridge this gap and propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions and generate solutions in a verifiable symbolic space. Using this framework as an analytical tool, we show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks compared to direct inference methods. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making. We propose two techniques to add trustworthy induction signals by generating familiar questions with similar underlying reasoning paths and asking models to perform self-refinement. Experiments show that our proposed techniques improve models' conceptual reasoning performance by 8% to 11%, achieving a more robust reasoning system that relies less on inductive biases.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 5 of its 28 references that are in the map. First posted on arXiv 30 Mar 2024.

Builds on

5 of 28 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners24 May

Other references · 4

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2024Mixtral of Expertscited by 1,785
  3. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  4. 2023System 2 Attention (is something you might need too)cited by 85

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 8; 2026: 1.

19 papers.

Sort:

By area:

  1. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  2. 2025ThinkGuard: Deliberative Slow Thinking Leads to Cautious Guardrailscited by 29
  3. 2024BIRD: A Trustworthy Bayesian Inference Framework for Large Language Modelscited by 18
  4. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  5. 2025One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMscited by 12
  6. 2024Abstraction-of-Thought Makes Language Models Better Reasonerscited by 11
  7. 2024ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoningcited by 9
  8. 2025AUTOCT: Automating Interpretable Clinical Trial Prediction with LLM Agentscited by 9
  9. 2024ComparisonQA: Evaluating Factuality Robustness of LLMs Through Knowledge Frequency Control and Uncertaintycited by 8
  10. 2025Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoningcited by 5
  11. 2024Concept-Reversed Winograd Schema Challenge: Evaluating and Improving Robust Reasoning in Large Language Models via Abstractioncited by 4
  12. 2025AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMscited by 4
7 more
  1. 2024ToW: Thoughts of Words Improve Reasoning in Large Language Modelscited by 2
  2. 2025BOW: Training Language Models to Reason Over Plausible Next Wordscited by 2
  3. 2024A Survey of Query Optimization in Large Language Modelscited by 1
  4. 2024Explicit Inductive Inference using Large Language Modelscited by 0
  5. 2025Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applicationscited by 0
  6. 2025CORE: Concept-Oriented Reinforcement for Bridging the Definition-Application Gap in Mathematical Reasoningcited by 0
  7. 2026The Presupposition Problem in Representation Genesiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  2. 2024Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
  3. 2022Towards Reasoning in Large Language Models: A Survey
  4. 2025Reasoning Capabilities and Invariability of Large Language Models
  5. 2026Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models
  6. 2023Hypothesis Search: Inductive Reasoning with Language Models
  7. 2023Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement
  8. 2024Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset