Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
Logical reasoning with large language models (LLMs) has received growing attention. One mainstream approach translates natural language into formal logic and then applies symbolic solvers for deduction. While effective in many tasks, these LLM-based translators often fail to generate consistent symbolic representations when the same concept appears in different linguistic forms. Such inconsistencies break logical coherence and lead to solver errors. However, most existing benchmarks lack this type of linguistic variation, which frequently occurs in real-world text, leaving the problem underexplored. To address this gap, we present SoLT, a benchmark that systematically rewrites reasoning datasets into diverse yet logically equivalent forms across multiple levels. Beyond evaluation, SoLT also provides a general method to enrich any dataset with linguistic diversity while preserving both meaning and logic. To further enhance the stability of LLM-based reasoning, we propose MenTaL, which explicitly guides models to build a concept-symbol mapping table during translation. By linking equivalent expressions to shared symbols, MenTaL maintains consistency and mitigates symbol drift. Experiments on SoLT demonstrate that LLMs indeed suffer from inconsistent symbol mapping under linguistic variation, leading to significant drops in reasoning accuracy. Meanwhile, applying MenTaL brings clear and stable performance improvements across diverse inputs. Overall, our findings reveal that overlooking linguistic diversity hides key weaknesses in LLM-based translators, and our work offers a step toward more reliable logical reasoning in varied real-world scenarios. Our code is available at https://github.com/wufeiwuwoshihua/LinguDiver.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 56 references that are in the map. First posted on arXiv 5 Jun 2025.
Builds on
23 of 56 references.
Closest ancestors · 9
Same direction, newest first
- 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation10 Feb
- 2024DeepSeek-V3 Technical Report27 Dec
- 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach29 Oct
- 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning10 Oct
- 2024LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations22 Jun
- 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers23 Oct
- 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
- 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
- 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep
Other references · 14
- 2023GPT-4 Technical Reportcited by 18,132
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
- 2024A StrongREJECT for Empty Jailbreakscited by 352
- 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
- 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
- 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
- 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 68
- 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
- 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
- 2025Can Slow-thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecastingcited by 19
2 more
Built on by
Citing papers per year: 2025: 3; 2026: 2.
5 papers.
- 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 28
- 2025From Hypothesis to Premises: LLM-based Backward Logical Reasoning with Selective Symbolic Translationcited by 2
- 2026Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Provingcited by 2
- 2026Semiotic logical hexagon theory for LLM logical reasoningcited by 1
- 2025PCRLLM: Proof-Carrying Reasoning with Large Language Models under Stepwise Logical Constraintscited by 0
- 2026Semiotic logical hexagon theory for LLM logical reasoning24 Jul 2026
- 2026Type-Checked Compliance: Deterministic Guardrails for Agentic Financial Systems Using Lean 4 Theorem Proving1 Apr 2026
- 2025From Hypothesis to Premises: LLM-based Backward Logical Reasoning with Selective Symbolic Translation3 Dec 2025
- 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learning18 Nov 2025
- 2025PCRLLM: Proof-Carrying Reasoning with Large Language Models under Stepwise Logical Constraints11 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
- 2024Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification
- 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
- 2026Improving Symbolic Translation of Language Models for Logical Reasoning
- 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
- 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
- 2024Can Large Language Models Act as Symbolic Reasoners?
- 2025Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning