mlsanityData to 7 Oct 2026

Are LLMs Stable Formal Logic Translators in Logical Reasoning Across Linguistically Diversified Texts?

arXiv 2506.04575 ↗First posted 5 Jun 2025cs.CLcited by 5

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Logical reasoning with large language models (LLMs) has received growing attention. One mainstream approach translates natural language into formal logic and then applies symbolic solvers for deduction. While effective in many tasks, these LLM-based translators often fail to generate consistent symbolic representations when the same concept appears in different linguistic forms. Such inconsistencies break logical coherence and lead to solver errors. However, most existing benchmarks lack this type of linguistic variation, which frequently occurs in real-world text, leaving the problem underexplored. To address this gap, we present SoLT, a benchmark that systematically rewrites reasoning datasets into diverse yet logically equivalent forms across multiple levels. Beyond evaluation, SoLT also provides a general method to enrich any dataset with linguistic diversity while preserving both meaning and logic. To further enhance the stability of LLM-based reasoning, we propose MenTaL, which explicitly guides models to build a concept-symbol mapping table during translation. By linking equivalent expressions to shared symbols, MenTaL maintains consistency and mitigates symbol drift. Experiments on SoLT demonstrate that LLMs indeed suffer from inconsistent symbol mapping under linguistic variation, leading to significant drops in reasoning accuracy. Meanwhile, applying MenTaL brings clear and stable performance improvements across diverse inputs. Overall, our findings reveal that overlooking linguistic diversity hides key weaknesses in LLM-based translators, and our work offers a step toward more reliable logical reasoning in varied real-world scenarios. Our code is available at https://github.com/wufeiwuwoshihua/LinguDiver.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 23 of its 56 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

23 of 56 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation10 Feb
  2. 2024DeepSeek-V3 Technical Report27 Dec
  3. 2024Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach29 Oct
  4. 2024Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning10 Oct
  5. 2024LOGIC-LM++: Multi-Step Refinement for Symbolic Formulations22 Jun
  6. 2023LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers23 Oct
  7. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning20 May
  8. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  9. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2024A StrongREJECT for Empty Jailbreakscited by 352
  6. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 309
  7. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  8. 2023An LLM can Fool Itself: A Prompt-Based Adversarial Attackcited by 111
  9. 2020LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoningcited by 68
  10. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
  11. 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
  12. 2025Can Slow-thinking LLMs Reason Over Time? Empirical Studies in Time Series Forecastingcited by 19
2 more
  1. 2024Quantifying perturbation impacts for large language modelscited by 8
  2. 2025am-ELO: A Stable Framework for Arena-based LLM Evaluationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Closer Look at Logical Reasoning with LLMs: The Choice of Tool Matters
  2. 2024Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification
  3. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  4. 2026Improving Symbolic Translation of Language Models for Logical Reasoning
  5. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  6. 2026Beyond Surface Forms: Symbolic Edits as a Test for Logical Reasoning with LLMs
  7. 2024Can Large Language Models Act as Symbolic Reasoners?
  8. 2025Adaptive Selection of Symbolic Languages for Improving LLM Logical Reasoning