mlsanityData to 7 Oct 2026

LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles

arXiv 2308.10855 ↗First posted 21 Aug 2023cs.CLcited by 41

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

With the continuous evolution and refinement of LLMs, they are endowed with impressive logical reasoning or vertical thinking capabilities. But can they think out of the box? Do they possess proficient lateral thinking abilities? Following the setup of Lateral Thinking Puzzles, we propose a novel evaluation benchmark, LatEval, which assesses the model's lateral thinking within an interactive framework. In our benchmark, we challenge LLMs with 2 aspects: the quality of questions posed by the model and the model's capability to integrate information for problem-solving. We find that nearly all LLMs struggle with employing lateral thinking during interactions. For example, even the most advanced model, GPT-4, exhibits the advantage to some extent, yet still maintain a noticeable gap when compared to human. This evaluation benchmark provides LLMs with a highly challenging and distinctive task that is crucial to an effective AI assistant.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 35 of its 58 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

35 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity8 Feb

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
22 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2023Tool Learning with Foundation Modelscited by 302
  5. 2021A Survey of Human-in-the-loop for Machine Learningcited by 214
  6. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  7. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  8. 2022A Survey on Retrieval-Augmented Text Generationcited by 157
  9. 2022Language Model Cascadescited by 104
  10. 2023Algorithm of Thoughts: Enhancing Exploration of Ideas in Large Language Modelscited by 92
  11. 2022Evaluating Human-Language Model Interactioncited by 90
  12. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  13. 2021Are we ready for a new paradigm shift? A Survey on Visual Deep MLPcited by 63
  14. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  15. 2022The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checkingcited by 56
  16. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 53
  17. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 48
  18. 2023Interactive Natural Language Processingcited by 31
  19. 2022Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correctioncited by 25
  20. 2023A Survey on Out-of-Distribution Evaluation of Neural NLP Modelscited by 20
  21. 2023MESED: A Multi-modal Entity Set Expansion Dataset with Fine-grained Semantic Classes and Hard Negative Entitiescited by 13
  22. 2022Towards Attribute-Entangled Controllable Text Generation: A Pilot Study of Blessing Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 16; 2025: 15; 2026: 8.

41 papers.

Sort:

By area:

  1. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  2. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  3. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  4. 2025Table-R1: Region-based Reinforcement Learning for Table Understandingcited by 35
  5. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  6. 2025Creative Preference Optimizationcited by 24
  7. 2024When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Modelscited by 22
  8. 2024Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmarkcited by 22
  9. 2025DAST: Context-Aware Compression in LLMs via Dynamic Allocation of Soft Tokenscited by 21
  10. 2024Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?cited by 18
  11. 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answeringcited by 18
  12. 2025Refine Knowledge of Large Language Models via Adaptive Contrastive Learningcited by 14
29 more
  1. 2024ProductAgent: Benchmarking Conversational Product Search Agent with Asking Clarification Questionscited by 10
  2. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10
  3. 2023EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Datacited by 9
  4. 2024Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correctioncited by 9
  5. 2024Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzlescited by 7
  6. 2024Creativity in AI: Progresses and Challengescited by 7
  7. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehuntscited by 7
  8. 2026EvoConfig: Self-Evolving Multi-Agent Systems for Efficient Autonomous Environment Configurationcited by 7
  9. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  10. 2024Exploring the Implicit Semantic Ability of Multimodal Large Language Models: A Pilot Study on Entity Set Expansioncited by 3
  11. 2025Solving Situation Puzzles with Large Language Model and External Reformulationcited by 3
  12. 2025Puzzled by Puzzles: When Vision-Language Models Can't Take a Hintcited by 3
  13. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 3
  14. 2024A Survey on Complex Tasks for Goal-Directed Interactive Agentscited by 2
  15. 2025Teaching According to Talents! Instruction Tuning LLMs with Competence-Aware Curriculum Learningcited by 2
  16. 2024MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughtscited by 1
  17. 2025Dynamic Reinforcement Learning for Actorscited by 1
  18. 2025Corrections Meet Explanations: A Unified Framework for Explainable Grammatical Error Correctioncited by 1
  19. 2025Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Taskscited by 1
  20. 2025Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Modelscited by 1
  21. 2024iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teaserscited by 0
  22. 2024BAMO at SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sensecited by 0
  23. 2025CL2GEC: A Multi-Discipline Benchmark for Continual Learning in Chinese Literature Grammatical Error Correctioncited by 0
  24. 2025PHANTOM RECALL: When Familiar Puzzles Fool Smart Modelscited by 0
  25. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  26. 2026IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMscited by 0
  27. 2026Can MLLMs Decode the Creative Leap? Introducing C4 for Cross-Concept Understandingcited by 0
  28. 2026From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilitiescited by 0
  29. 2026AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024uTeBC-NLP at SemEval-2024 Task 9: Can LLMs be Lateral Thinkers?
  2. 2025Multi-Turn Puzzles: Evaluating Interactive Reasoning and Strategic Dialogue in LLMs
  3. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  4. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
  5. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  6. 2026Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework
  7. 2026UniCog: Uncovering Cognitive Abilities of LLMs through Latent Mind Space Analysis
  8. 2024MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs