MasonTigers at SemEval-2024 Task 9: Solving Puzzles with an Ensemble of Chain-of-Thoughts
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
Our paper presents team MasonTigers submission to the SemEval-2024 Task 9 - which provides a dataset of puzzles for testing natural language understanding. We employ large language models (LLMs) to solve this task through several prompting techniques. Zero-shot and few-shot prompting generate reasonably good results when tested with proprietary LLMs, compared to the open-source models. We obtain further improved results with chain-of-thought prompting, an iterative prompting method that breaks down the reasoning process step-by-step. We obtain our best results by utilizing an ensemble of chain-of-thought prompts, placing 2nd in the word puzzle subtask and 13th in the sentence puzzle subtask. The strong performance of prompted LLMs demonstrates their capability for complex reasoning when provided with a decomposition of the thought process. Our work sheds light on how step-wise explanatory prompts can unlock more of the knowledge encoded in the parameters of large models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 19 references that are in the map. First posted on arXiv 22 Mar 2024.
Builds on
8 of 19 references.
References · 8
- 2023GPT-4 Technical Reportcited by 18,132
- 2021Program Synthesis with Large Language Modelscited by 3,509
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
- 2024Mixtral of Expertscited by 1,785
- 2023LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzlescited by 41
- 2023ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sensecited by 27
- 2023Tree of Uncertain Thoughts Reasoning for Large Language Modelscited by 19
- 2020CS-NLP team at SemEval-2020 Task 4: Evaluation of State-of-the-art NLP Deep Learning Architectures on Commonsense Reasoning Taskcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- 2023An automatically discovered chain-of-thought prompt generalizes to novel models and datasets
- 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
- 2024Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns
- 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods
- 2024Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation
- 2024Demystifying Chains, Trees, and Graphs of Thoughts