mlsanityData to 7 Oct 2026

BRAINTEASER: Lateral Thinking Puzzles for Large Language Models

arXiv 2310.05057 ↗First posted 8 Oct 2023cs.CLcited by 18

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

The success of language models has inspired the NLP community to attend to tasks that require implicit and complex reasoning, relying on human-like commonsense mechanisms. While such vertical thinking tasks have been relatively popular, lateral thinking puzzles have received little attention. To bridge this gap, we devise BRAINTEASER: a multiple-choice Question Answering task designed to test the model's ability to exhibit lateral thinking and defy default commonsense associations. We design a three-step procedure for creating the first lateral thinking benchmark, consisting of data collection, distractor generation, and generation of adversarial examples, leading to 1,100 puzzles with high-quality annotations. To assess the consistency of lateral reasoning by models, we enrich BRAINTEASER based on a semantic and contextual reconstruction of its questions. Our experiments with state-of-the-art instruction- and commonsense language models reveal a significant gap between human and model performance, which is further widened when consistency across adversarial formats is considered. We make all of our code and data available to stimulate work on developing and evaluating lateral thinking models.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 55 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

39 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2022Emergent Abilities of Large Language Modelscited by 2,483
26 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  10. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  11. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  12. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  13. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  14. 2023On the Creativity of Large Language Modelscited by 57
  15. 2020ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoningcited by 45
  16. 2020CSKG: The CommonSense Knowledge Graphcited by 45
  17. 2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
  18. 2019Towards Generalizable Neuro-Symbolic Systems for Commonsense Question Answeringcited by 37
  19. 2018Neural Metaphor Detection in Contextcited by 34
  20. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  21. 2021Dimensions of Commonsense Knowledgecited by 29
  22. 2023CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answeringcited by 28
  23. 2020The Sensitivity of Language Models and Humans to Winograd Schema Perturbationscited by 21
  24. 2021Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Modelscited by 11
  25. 2019Joint Detection and Location of English Punscited by 9
  26. 2023Two Kinds of Recallcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 9; 2026: 4.

18 papers.

Sort:

By area:

  1. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  2. 2024Optimizing Language Model's Reasoning Abilities with Weak Supervisioncited by 15
  3. 2024HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Modelscited by 15
  4. 2025LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactionscited by 14
  5. 2025A Causality-aware Paradigm for Evaluating Creativity of Multimodal Large Language Modelscited by 10
  6. 2024Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzlescited by 7
  7. 2025From Reasoning to Learning: A Survey on Hypothesis Discovery and Rule Learning with Large Language Modelscited by 5
  8. 2025Solving Situation Puzzles with Large Language Model and External Reformulationcited by 3
  9. 2024Aligning Generalisation Between Humans and Machinescited by 2
  10. 2025Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Modelscited by 2
  11. 2025Dynamic Reinforcement Learning for Actorscited by 1
  12. 2025Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Taskscited by 1
6 more
  1. 2025Reasoning Riddles: How Explainability Reveals Cognitive Limits in Vision-Language Modelscited by 1
  2. 2025AssoCiAm: A Benchmark for Evaluating Association Thinking while Circumventing Ambiguitycited by 0
  3. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  4. 2026AGC-Bench: Measuring Artificial General Creativitycited by 0
  5. 2026Can MLLMs Decode the Creative Leap? Introducing C4 for Cross-Concept Understandingcited by 0
  6. 2026AhaBench: Do Agents Turn Experience into Reusable Insights? A Long-Horizon Benchmark for Continual Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Augmenting Lateral Thinking in Language Models with Humor and Riddle Data for the BRAINTEASER Task
  2. 2024SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense
  3. 2024Missed Connections: Lateral Thinking Puzzles for Large Language Models
  4. 2026BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models
  5. 2024iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers
  6. 2024BAMO at SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense
  7. 2024AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles
  8. 2024Mothman at SemEval-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization