mlsanityData to 7 Oct 2026

AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles

arXiv 2404.01084 ↗First posted 1 Apr 2024cs.CL cs.AIcited by 1

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

In this paper, we outline our submission for the SemEval-2024 Task 9 competition: 'BRAINTEASER: A Novel Task Defying Common Sense'. We engage in both sub-tasks: Sub-task A-Sentence Puzzle and Sub-task B-Word Puzzle. We evaluate a plethora of pre-trained transformer-based language models of different sizes through fine-tuning. Subsequently, we undertake an analysis of their scores and responses to aid future researchers in understanding and utilizing these models effectively. Our top-performing approaches secured competitive positions on the competition leaderboard across both sub-tasks. In the evaluation phase, our best submission attained an average accuracy score of 81.7% in the Sentence Puzzle, and 85.4% in the Word Puzzle, significantly outperforming the best neural baseline (ChatGPT) by more than 20% and 30% respectively.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 43 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

27 of 43 references.

References · 27

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2023Mistral 7Bcited by 3,065
  7. 2022Emergent Abilities of Large Language Modelscited by 2,483
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2024Mixtral of Expertscited by 1,785
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  12. 2023Textbooks Are All You Needcited by 561
15 more
  1. 2023How Language Model Hallucinations Can Snowballcited by 310
  2. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  3. 2023LLMs cannot find reasoning errors, but can correct them given the error locationcited by 85
  4. 2023A Survey of Reasoning with Foundation Modelscited by 70
  5. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  6. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 60
  7. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  8. 2023Logical Reasoning over Natural Language as Knowledge Representation: A Surveycited by 32
  9. 2016Recovery guarantee of weighted low-rank approximation via alternating minimizationcited by 20
  10. 2023tasksource: A Dataset Harmonization Framework for Streamlined NLP Multi-Task Learning and Evaluationcited by 11
  11. 2023Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Frameworkcited by 11
  12. 2023Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performancecited by 9
  13. 2017Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activationscited by 8
  14. 2023Commonsense Reasoning for Conversational AI: A Survey of the State of the Artcited by 7
  15. 2023A Survey of Methods, Challenges and Perspectives in Causalitycited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sensecited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2024BAMO at SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense
  2. 2024Augmenting Lateral Thinking in Language Models with Humor and Riddle Data for the BRAINTEASER Task
  3. 2024iREL at SemEval-2024 Task 9: Improving Conventional Prompting Methods for Brain Teasers
  4. 2023BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
  5. 2024MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts
  6. 2025Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
  7. 2024Missed Connections: Lateral Thinking Puzzles for Large Language Models
  8. 2026Neural Activation Patterns Across Language Model Architectures: A Comprehensive Analysis of Cognitive Task Performance