# AILS-NTUA at SemEval-2024 Task 9: Cracking Brain Teasers: Transformer Models for Lateral Thinking Puzzles

arXiv: https://arxiv.org/abs/2404.01084 · HTML: https://mlsanity.com/paper/2404.01084/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 1 Apr 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 27 of 27)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [WinoGrande: An Adversarial Winograd Schema Challenge at Scale](https://mlsanity.com/paper/1907.10641/) (2019)
- [LLMs cannot find reasoning errors, but can correct them given the error location](https://mlsanity.com/paper/2311.08516/) (2023)
- [A Survey of Reasoning with Foundation Models](https://mlsanity.com/paper/2312.11562/) (2023)
- [Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian](https://mlsanity.com/paper/1906.05392/) (2019)
- [RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledge](https://mlsanity.com/paper/2101.00376/) (2021)
- [Puzzle Solving using Reasoning of Large Language Models: A Survey](https://mlsanity.com/paper/2402.11291/) (2024)
- [Logical Reasoning over Natural Language as Knowledge Representation: A Survey](https://mlsanity.com/paper/2303.12023/) (2023)
- [Recovery guarantee of weighted low-rank approximation via alternating minimization](https://mlsanity.com/paper/1602.02262/) (2016)
- [tasksource: A Dataset Harmonization Framework for Streamlined NLP Multi-Task Learning and Evaluation](https://mlsanity.com/paper/2301.05948/) (2023)
- [Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework](https://mlsanity.com/paper/2307.12626/) (2023)
- [Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance](https://mlsanity.com/paper/2310.05597/) (2023)
- [Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations](https://mlsanity.com/paper/1712.09203/) (2017)
- [Commonsense Reasoning for Conversational AI: A Survey of the State of the Art](https://mlsanity.com/paper/2302.07926/) (2023)
- [A Survey of Methods, Challenges and Perspectives in Causality](https://mlsanity.com/paper/2302.00293/) (2023)

## Built on by (most cited first, 1 of 1)

- [SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense](https://mlsanity.com/paper/2404.16068/) (2024)

## Abstract (arXiv)

In this paper, we outline our submission for the SemEval-2024 Task 9 competition: 'BRAINTEASER: A Novel Task Defying Common Sense'. We engage in both sub-tasks: Sub-task A-Sentence Puzzle and Sub-task B-Word Puzzle. We evaluate a plethora of pre-trained transformer-based language models of different sizes through fine-tuning. Subsequently, we undertake an analysis of their scores and responses to aid future researchers in understanding and utilizing these models effectively. Our top-performing approaches secured competitive positions on the competition leaderboard across both sub-tasks. In the evaluation phase, our best submission attained an average accuracy score of 81.7% in the Sentence Puzzle, and 85.4% in the Word Puzzle, significantly outperforming the best neural baseline (ChatGPT) by more than 20% and 30% respectively.