# Missed Connections: Lateral Thinking Puzzles for Large Language Models

arXiv: https://arxiv.org/abs/2404.11730 · HTML: https://mlsanity.com/paper/2404.11730/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Emergent Analogical Reasoning in Large Language Models](https://mlsanity.com/paper/2212.09196/) (2022)
- [Learning to Speak and Act in a Fantasy Text Adventure Game](https://mlsanity.com/paper/1903.03094/) (2019)
- [Level Generation Through Large Language Models](https://mlsanity.com/paper/2302.05817/) (2023)
- [MarioGPT: Open-Ended Text2Level Generation through Large Language Models](https://mlsanity.com/paper/2302.05981/) (2023)
- [Super Mario as a String: Platformer Level Generation Via LSTMs](https://mlsanity.com/paper/1603.00930/) (2016)
- [The Chess Transformer: Mastering Play using Generative Language Models](https://mlsanity.com/paper/2008.04057/) (2020)
- [Can Large Language Models Play Text Games Well? Current State-of-the-Art and Open Questions](https://mlsanity.com/paper/2304.02868/) (2023)
- [ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Games](https://mlsanity.com/paper/2305.14879/) (2023)
- [Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP](https://mlsanity.com/paper/2104.08620/) (2021)
- [The Go Transformer: Natural Language Modeling for Game Play](https://mlsanity.com/paper/2007.03500/) (2020)
- [Solving and Generating NPR Sunday Puzzles with Large Language Models](https://mlsanity.com/paper/2306.12255/) (2023)

## Built on by (most cited first, 9 of 9)

- [BALROG: Benchmarking Agentic LLM and VLM Reasoning On Games](https://mlsanity.com/paper/2411.13543/) (2024)
- [Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples](https://mlsanity.com/paper/2406.05673/) (2024)
- [Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles](https://mlsanity.com/paper/2410.06733/) (2024)
- [ChatPCG: Large Language Model-Driven Reward Design for Procedural Content Generation](https://mlsanity.com/paper/2406.11875/) (2024)
- [NYT-Connections: A Deceptively Simple Text Classification Task that Stumps System-1 Thinkers](https://mlsanity.com/paper/2412.01621/) (2024)
- [Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game](https://mlsanity.com/paper/2406.11012/) (2024)
- [Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian Rebuses](https://mlsanity.com/paper/2408.00584/) (2024)
- [LLMs as Method Actors: A Model for Prompt Engineering and Architecture](https://mlsanity.com/paper/2411.05778/) (2024)
- [CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge](https://mlsanity.com/paper/2604.03374/) (2026)

## Abstract (arXiv)

The Connections puzzle published each day by the New York Times tasks players with dividing a bank of sixteen words into four groups of four words that each relate to a common theme. Solving the puzzle requires both common linguistic knowledge (i.e. definitions and typical usage) as well as, in many cases, lateral or abstract thinking. This is because the four categories ascend in complexity, with the most challenging category often requiring thinking about words in uncommon ways or as parts of larger phrases. We investigate the capacity for automated AI systems to play Connections and explore the game's potential as an automated benchmark for abstract reasoning and a way to measure the semantic information encoded by data-driven linguistic systems. In particular, we study both a sentence-embedding baseline and modern large language models (LLMs). We report their accuracy on the task, measure the impacts of chain-of-thought prompting, and discuss their failure modes. Overall, we find that the Connections task is challenging yet feasible, and a strong test-bed for future work.