mlsanityData to 14 Oct 2026

Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Tasks

arXiv 2505.23843 ↗First posted 28 May 2025cs.CL cs.LGcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Multi-round incomplete information tasks are crucial for evaluating the lateral thinking capabilities of large language models (LLMs). Currently, research primarily relies on multiple benchmarks and automated evaluation metrics to assess these abilities. However, our study reveals novel insights into the limitations of existing methods, as they often yield misleading results that fail to uncover key issues, such as shortcut-taking behaviors, rigid patterns, and premature task termination. These issues obscure the true reasoning capabilities of LLMs and undermine the reliability of evaluations. To address these limitations, we propose a refined set of evaluation standards, including inspection of reasoning paths, diversified assessment metrics, and comparative analyses with human performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 33 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

20 of 33 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Solving Situation Puzzles with Large Language Model and External Reformulation24 Mar
  2. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks19 Dec
  3. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels4 Sep
  4. 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles21 Aug

Other references · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  4. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 66
  5. 2023Probing the Creativity of Large Language Models: Can models produce divergent semantic association?cited by 33
  6. 2024MathChat: Benchmarking Mathematical Reasoning and Instruction Following in Multi-Turn Interactionscited by 26
  7. 2024IdeaBench: Benchmarking Large Language Models for Research Idea Generationcited by 23
  8. 2024Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Modelscited by 20
  9. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Modelscited by 18
  10. 2024Lost in the Middle, and In-Between: Enhancing Language Models' Ability to Reason Over Long Contexts in Multi-Hop QAcited by 13
  11. 2024Derailer-Rerailer: Adaptive Verification for Efficient and Reliable Language Model Reasoningcited by 9
  12. 2024Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzlescited by 8
4 more
  1. 2025TextGames: Learning to Self-Play Text-Based Puzzle Games via Language Model Reasoningcited by 8
  2. 2024TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzlescited by 7
  3. 2024FSM: A Finite State Machine Based Zero-Shot Prompting Paradigm for Multi-Hop Question Answeringcited by 6
  4. 2024Unleashing Multi-Hop Reasoning Potential in Large Language Models through Repetition of Misordered Contextcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ZPD-SCA: Unveiling the Blind Spots of LLMs in Assessing Students' Cognitive Abilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  2. 2023Mind's Mirror: Distilling Self-Evaluation Capability and Comprehensive Thinking from Large Language Models
  3. 2024Enhancing LLM Evaluations: The Garbling Trick
  4. 2025Position: On the Methodological Pitfalls of Evaluating Base LLMs for Reasoning
  5. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  6. 2025Beyond Chains of Thought: Benchmarking Latent-Space Reasoning Abilities in Large Language Models
  7. 2023Evaluating the Deductive Competence of Large Language Models
  8. 2026LLM Reasoning Is Latent, Not the Chain of Thought