# MacGyver: Are Large Language Models Creative Problem Solvers?

arXiv: https://arxiv.org/abs/2311.09682 · HTML: https://mlsanity.com/paper/2311.09682/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference](https://mlsanity.com/paper/1808.05326/) (2018)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [PHYRE: A New Benchmark for Physical Reasoning](https://mlsanity.com/paper/1908.05656/) (2019)
- [We're Afraid Language Models Aren't Modeling Ambiguity](https://mlsanity.com/paper/2304.14399/) (2023)
- [Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contest](https://mlsanity.com/paper/2209.06293/) (2022)
- [Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writing](https://mlsanity.com/paper/2210.13669/) (2022)
- [Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks](https://mlsanity.com/paper/2205.05718/) (2022)
- [PROST: Physical Reasoning of Objects through Space and Time](https://mlsanity.com/paper/2106.03634/) (2021)
- [NEWTON: Are Large Language Models Capable of Physical Reasoning?](https://mlsanity.com/paper/2310.07018/) (2023)
- [PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning](https://mlsanity.com/paper/2112.05136/) (2021)
- [Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers](https://mlsanity.com/paper/2309.12570/) (2023)
- [AmbiPun: Generating Humorous Puns with Ambiguous Context](https://mlsanity.com/paper/2205.01825/) (2022)

## Built on by (most cited first, 30 of 34)

- [SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs](https://mlsanity.com/paper/2410.13648/) (2024)
- [Divergent Creativity in Humans and Large Language Models](https://mlsanity.com/paper/2405.13012/) (2024)
- [LLMPhy: Parameter-Identifiable Physical Reasoning Combining Large Language Models and Physics Engines](https://mlsanity.com/paper/2411.08027/) (2024)
- [Evaluating Creative Short Story Generation in Humans and Large Language Models](https://mlsanity.com/paper/2411.02316/) (2024)
- [Eureka: Evaluating and Understanding Large Foundation Models](https://mlsanity.com/paper/2409.10566/) (2024)
- [LLMs Can't Handle Peer Pressure: Crumbling under Multi-Agent Social Interactions](https://mlsanity.com/paper/2508.18321/) (2025)
- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [Creative Problem Solving in Large Language and Vision Models -- What Would it Take?](https://mlsanity.com/paper/2405.01453/) (2024)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints](https://mlsanity.com/paper/2606.05622/) (2026)
- [Characterising the Creative Process in Humans and Large Language Models](https://mlsanity.com/paper/2405.00899/) (2024)
- [Measuring LLM Novelty As The Frontier Of Original And High-Quality Output](https://mlsanity.com/paper/2504.09389/) (2025)
- [GODBench: A Benchmark for Multimodal Large Language Models in Video Comment Art](https://mlsanity.com/paper/2505.11436/) (2025)
- [The creative psychometric item generator: a framework for item generation and validation using large language models](https://mlsanity.com/paper/2409.00202/) (2024)
- [Transferring Natural Language Datasets Between Languages Using Large Language Models for Modern Decision Support and Sci-Tech Analytical Systems](https://mlsanity.com/paper/2410.14074/) (2024)
- [To Trade or Not to Trade: An Agentic Approach to Estimating Market Risk Improves Trading Decisions](https://mlsanity.com/paper/2507.08584/) (2025)
- [What Shapes a Creative Machine Mind? Comprehensively Benchmarking Creativity in Foundation Models](https://mlsanity.com/paper/2510.04009/) (2025)
- [Are LLMs becoming similarly creative? Evidence from three years of models](https://mlsanity.com/paper/2608.19437/) (2026)
- [GuessBench: Sensemaking Multimodal Creativity in the Wild](https://mlsanity.com/paper/2506.00814/) (2025)
- [A Study of Commonsense Reasoning over Visual Object Properties](https://mlsanity.com/paper/2508.10956/) (2025)
- [On the Creativity of AI Agents](https://mlsanity.com/paper/2604.13242/) (2026)
- [RoboWits: Unexpected Challenges for Robotic Creative Problem Solving](https://mlsanity.com/paper/2605.30326/) (2026)
- [Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks](https://mlsanity.com/paper/2608.30047/) (2026)
- [Abductive Computational Systems: Creative Abduction and Future Directions](https://mlsanity.com/paper/2507.08264/) (2025)
- [ComboBench: Can LLMs Manipulate Physical Devices to Play Virtual Reality Games?](https://mlsanity.com/paper/2510.24706/) (2025)
- [Universe of Thoughts: Enabling Creative Reasoning with Large Language Models](https://mlsanity.com/paper/2511.20471/) (2025)
- [CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledge](https://mlsanity.com/paper/2604.03374/) (2026)
- [Large Language Models Align with the Human Brain during Creative Thinking](https://mlsanity.com/paper/2604.03480/) (2026)
- [How Creative Are Large Language Models in Generating Molecules?](https://mlsanity.com/paper/2604.18031/) (2026)
- [IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs](https://mlsanity.com/paper/2606.00875/) (2026)

## Abstract (arXiv)

We explore the creative problem-solving capabilities of modern LLMs in a novel constrained setting. To this end, we create MACGYVER, an automatically generated dataset consisting of over 1,600 real-world problems deliberately designed to trigger innovative usage of objects and necessitate out-of-the-box thinking. We then present our collection to both LLMs and humans to compare and contrast their problem-solving abilities. MACGYVER is challenging for both groups, but in unique and complementary ways. For instance, humans excel in tasks they are familiar with but struggle with domain-specific knowledge, leading to a higher variance. In contrast, LLMs, exposed to a variety of specialized knowledge, attempt broader problems but fail by proposing physically-infeasible actions. Finally, we provide a detailed error analysis of LLMs, and demonstrate the potential of enhancing their problem-solving ability with novel prompting techniques such as iterative step-wise reflection and divergent-convergent thinking. This work (1) introduces a fresh arena for intelligent agents focusing on intricate aspects of physical reasoning, planning, and unconventional thinking, which supplements the existing spectrum of machine intelligence; and (2) provides insight into the constrained problem-solving capabilities of both humans and AI.