# Connecting the Dots: Evaluating Abstract Reasoning Capabilities of LLMs Using the New York Times Connections Word Game

arXiv: https://arxiv.org/abs/2406.11012 · HTML: https://mlsanity.com/paper/2406.11012/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 14 of its 43 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 14 of 14)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [On the Measure of Intelligence](https://mlsanity.com/paper/1911.01547/) (2019)
- [SmartPlay: A Benchmark for LLMs as Intelligent Agents](https://mlsanity.com/paper/2310.01557/) (2023)
- [How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments](https://mlsanity.com/paper/2403.11807/) (2024)
- [GameGPT: Multi-agent Collaborative Framework for Game Development](https://mlsanity.com/paper/2310.08067/) (2023)
- [The Chess Transformer: Mastering Play using Generative Language Models](https://mlsanity.com/paper/2008.04057/) (2020)
- [GameEval: Evaluating LLMs on Conversational Games](https://mlsanity.com/paper/2308.10032/) (2023)
- [Fast and flexible: Human program induction in abstract reasoning tasks](https://mlsanity.com/paper/2103.05823/) (2021)
- [Game Generation via Large Language Models](https://mlsanity.com/paper/2404.08706/) (2024)
- [Modeling Worlds in Text](https://mlsanity.com/paper/2106.09578/) (2021)
- [Missed Connections: Lateral Thinking Puzzles for Large Language Models](https://mlsanity.com/paper/2404.11730/) (2024)
- [Solving and Generating NPR Sunday Puzzles with Large Language Models](https://mlsanity.com/paper/2306.12255/) (2023)
- [A Unified Feature Representation for Lexical Connotations](https://mlsanity.com/paper/2006.00635/) (2020)

## Built on by (most cited first, 3 of 3)

- [GeoExplain: Multimodal Reasoning based on Hierarchy of Visual Information in Street View](https://mlsanity.com/paper/2506.16633/) (2025)
- [LLMs as Method Actors: A Model for Prompt Engineering and Architecture](https://mlsanity.com/paper/2411.05778/) (2024)
- [Think Globally, Group Locally: Evaluating LLMs Using Multi-Lingual Word Grouping Games](https://mlsanity.com/paper/2510.14030/) (2025)

## Abstract (arXiv)

The New York Times Connections game has emerged as a popular and challenging pursuit for word puzzle enthusiasts. We collect 438 Connections games to evaluate the performance of state-of-the-art large language models (LLMs) against expert and novice human players. Our results show that even the best performing LLM, Claude 3.5 Sonnet, which has otherwise shown impressive reasoning abilities on a wide variety of benchmarks, can only fully solve 18% of the games. Novice and expert players perform better than Claude 3.5 Sonnet, with expert human players significantly outperforming it. We create a taxonomy of the knowledge types required to successfully cluster and categorize words in the Connections game. We find that while LLMs perform relatively well on categorizing words based on semantic relations they struggle with other types of knowledge such as Encyclopedic Knowledge, Multiword Expressions or knowledge that combines both Word Form and Meaning. Our results establish the New York Times Connections game as a challenging benchmark for evaluating abstract reasoning capabilities in AI systems.