mlsanityData to 7 Oct 2026

FRoG: Evaluating Fuzzy Reasoning of Generalized Quantifiers in Large Language Models

arXiv 2407.01046 ↗First posted 1 Jul 2024cs.AI cs.CLcited by 0

On the map: LLM reasoning → Mathematical reasoning

Abstract (arXiv)

Fuzzy reasoning is vital due to the frequent use of imprecise information in daily contexts. However, the ability of current large language models (LLMs) to handle such reasoning remains largely uncharted. In this paper, we introduce a new benchmark, FRoG, for fuzzy reasoning, featuring real-world mathematical word problems that incorporate generalized quantifiers. Our experimental findings reveal that fuzzy reasoning continues to pose significant challenges for LLMs. Moreover, we find that existing methods designed to enhance reasoning do not consistently improve performance in tasks involving fuzzy logic. Additionally, our results show an inverse scaling effect in the performance of LLMs on FRoG. Interestingly, we also demonstrate that strong mathematical reasoning skills are not necessarily indicative of success on our benchmark.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Mathematical reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 67 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

37 of 67 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Evaluating Mathematical Reasoning Beyond Accuracy8 Apr
  2. 2024Dual Instruction Tuning with Large Language Models for Mathematical Reasoning27 Mar
  3. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoning5 Mar
  4. 2024Key-Point-Driven Data Synthesis with its Enhancement on Mathematical Reasoning4 Mar
  5. 2024MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data14 Feb
  6. 2023Learning From Mistakes Makes LLM Better Reasoner31 Oct
  7. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text10 Oct
  8. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving29 Sep
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct18 Aug
  10. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification15 Aug
  11. 2022Lila: A Unified Benchmark for Mathematical Reasoning31 Oct

Other references · 26

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023Qwen Technical Reportcited by 3,273
  5. 2023Mistral 7Bcited by 3,065
  6. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  8. 2022Galactica: A Large Language Model for Sciencecited by 736
  9. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  10. 2024Yi: Open Foundation Models by 01.AIcited by 706
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
14 more
  1. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  2. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  4. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 225
  5. 2023Implicit Chain of Thought Reasoning via Knowledge Distillationcited by 165
  6. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  7. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  8. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  9. 2023Contrastive Chain-of-Thought Promptingcited by 52
  10. 2024Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategiescited by 44
  11. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Modelscited by 33
  12. 2024Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large Language Models Reasoningcited by 31
  13. 2024Fine-tuning Large Language Models with Sequential Instructionscited by 20
  14. 2022A Fundamental Probabilistic Fuzzy Logic Framework Suitable for Causal Reasoningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges
  2. 2026Mathematical Reasoning in Large Language Models: Benchmarks, Architectures, Evaluation, and Open Challenges
  3. 2025Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models
  4. 2026QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation
  5. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  6. 2025I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Models
  7. 2025Do Large Language Models Excel in Complex Logical Reasoning with Formal Language?
  8. 2022Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks