mlsanityData to 14 Oct 2026

Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset

arXiv 2306.11167 ↗First posted 19 Jun 2023cs.CL cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

The quest for human imitative AI has been an enduring topic in AI research since its inception. The technical evolution and emerging capabilities of the latest cohort of large language models (LLMs) have reinvigorated the subject beyond academia to the cultural zeitgeist. While recent NLP evaluation benchmark tasks test some aspects of human-imitative behaviour (e.g., BIG-bench's 'human-like behavior' tasks), few, if not none, examine creative problem solving abilities. Creative problem solving in humans is a well-studied topic in cognitive neuroscience with standardized tests that predominantly use the ability to associate (heterogeneous) connections among clue words as a metric for creativity. Exposure to misleading stimuli - distractors dubbed red herrings - impede human performance in such tasks via the fixation effect and Einstellung paradigm. In cognitive neuroscience studies, such fixations are experimentally induced by pre-exposing participants to orthographically similar incorrect words to subsequent word-fragments or clues. The popular British quiz show Only Connect's Connecting Wall segment essentially mimics Mednick's Remote Associates Test (RAT) formulation with built-in, deliberate red herrings, which makes it an ideal proxy dataset to explore and study fixation effect and Einstellung paradigm from cognitive neuroscience in LLMs. In this paper we present the novel Only Connect Wall (OCW) dataset and report results from our evaluation of selected pre-trained language models and LLMs on creative problem solving tasks like grouping clue words by heterogeneous connections, and identifying correct open knowledge domain connections in respective groups. We synthetically generate two additional datasets: OCW-Randomized, OCW-WordNet to further analyze our red-herrings hypothesis in language models. The code and link to the dataset are available at https://github.com/TaatiTeam/OCW.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 47 of its 94 references that are in the map. First posted on arXiv 19 Jun 2023.

Builds on

47 of 94 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  10. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2020Measuring Massive Multitask Language Understandingcited by 7,963
35 more
  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  10. 2023A Survey of Large Language Modelscited by 2,576
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2018Deep contextualized word representationscited by 2,290
  13. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  14. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  15. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  16. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  17. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  18. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  19. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  20. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  21. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  22. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  23. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  24. 2018Learning Word Vectors for 157 Languagescited by 280
  25. 2018Characterizing and Avoiding Negative Transfercited by 229
  26. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  27. 2015On Wasserstein Two Sample Testing and Related Families of Nonparametric Testscited by 166
  28. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  29. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 142
  30. 2014A Tutorial on Principal Component Analysiscited by 141
  31. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
  32. 2023Large Language Models can be Guided to Evade AI-Generated Text Detectioncited by 44
  33. 2022American == White in Multimodal Language-and-Image AIcited by 39
  34. 2023What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventoriescited by 9
  35. 2023GlobalBench: A Benchmark for Global Progress in Natural Language Processingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 6; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  2. 2025Limitations of Large Language Models in Clinical Problem-Solving Arising from Inflexible Reasoningcited by 31
  3. 2025Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluatorcited by 10
  4. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  5. 2024Creative Problem Solving in Large Language and Vision Models -- What Would it Take?cited by 7
  6. 2024Creativity in AI: Progresses and Challengescited by 7
  7. 2025Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Modelscited by 7
  8. 2023Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Modelscited by 6
  9. 2025Evaluating the Creativity of LLMs in Persian Literary Text Generationcited by 4
  10. 2026MedEinst: Benchmarking the Einstellung Effect in Medical LLMs through Counterfactual Differential Diagnosiscited by 4
  11. 2025DEUCE: Dual-diversity Enhancement and Uncertainty-awareness for Cold-start Active Learningcited by 2
  12. 2025GuessBench: Sensemaking Multimodal Creativity in the Wildcited by 1
5 more
  1. 2026Beyond Perception Errors: Semantic Fixation in Large Vision-Language Modelscited by 1
  2. 2026Entropic Context Shaping: Information-Theoretic Filtering for Context-Aware LLM Agentscited by 0
  3. 2026Advances in LLM Reasoning Enable Flexibility in Clinical Problem-Solvingcited by 0
  4. 2026CresOWLve: Benchmarking Creative Problem-Solving Over Real-World Knowledgecited by 0
  5. 2026Assessing the Creativity of Large Language Models: Testing, Limits, and New Frontierscited by 0