mlsanityData to 7 Oct 2026

The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding

arXiv 2502.08946 ↗First posted 13 Feb 2025cs.CL cs.AI cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

In a systematic way, we investigate a widely asked question: Do LLMs really understand what they say?, which relates to the more familiar term Stochastic Parrot. To this end, we propose a summative assessment over a carefully designed physical concept understanding task, PhysiCo. Our task alleviates the memorization issue via the usage of grid-format inputs that abstractly describe physical phenomena. The grids represents varying levels of understanding, from the core phenomenon, application examples to analogies to other abstract patterns in the grid world. A comprehensive study on our task demonstrates: (1) state-of-the-art LLMs, including GPT-4o, o1 and Gemini 2.0 flash thinking, lag behind humans by ~40%; (2) the stochastic parrot phenomenon is present in LLMs, as they fail on our grid task but can describe and recognize the same concepts well in natural language; (3) our task challenges the LLMs due to intrinsic difficulties rather than the unfamiliar grid format, as in-context learning and fine-tuning on same formatted data added little to their performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 13 Feb 2025.

Builds on

37 of 52 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2017Mask R-CNNcited by 7,259
  7. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2023Mistral 7Bcited by 3,065
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
25 more
  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  5. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  6. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  7. 2023Large Language Models as General Pattern Machinescited by 196
  8. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  9. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  10. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 119
  11. 2022Do Androids Laugh at Electric Sheep? Humor "Understanding" Benchmarks from The New Yorker Caption Contestcited by 98
  12. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 81
  13. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  14. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 78
  15. 2021Communicating Natural Programs to Humans and Machinescited by 35
  16. 2024Metaphor Understanding Challenge Dataset for LLMscited by 30
  17. 2023The Dark Side of ChatGPT: Legal and Ethical Challenges from Stochastic Parrots and Hallucinationcited by 23
  18. 2023Personality Understanding of Fictional Characters during Book Readingcited by 20
  19. 2023ANPL: Towards Natural Programming with Interactive Decompositioncited by 13
  20. 2023Large Language Model (LLM) as a System of Multiple Expert Agents: An Approach to solve the Abstraction and Reasoning Corpus (ARC) Challengecited by 11
  21. 2024Can LLMs Understand Computer Networks? Towards a Virtual System Administratorcited by 11
  22. 2023Stochastic Parrots Looking for Stochastic Parrots: LLMs are Easy to Fine-Tune and Hard to Detect with other LLMscited by 8
  23. 2024Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Modelscited by 4
  24. 2025Understanding LLMs' Fluid Intelligence Deficiency: An Analysis of the ARC Taskcited by 4
  25. 2024Previously on the Stories: Recap Snippet Identification for Story Readingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  2. 2025PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contextscited by 7
  3. 2025Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluationcited by 6
  4. 2026Many-Shot CoT-ICL: Making In-Context Learning Truly Learncited by 3
  5. 2025Ref-Long: Benchmarking the Long-context Referencing Capability of Long-context Language Modelscited by 2
  6. 2025InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoningcited by 2
  7. 2025Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMscited by 2
  8. 2025Beyond Mimicry: Preference Coherence in LLMscited by 2
  9. 2026ClimAgent: LLM as Agents for Autonomous Open-ended Climate Science Analysiscited by 1
  10. 2025GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Modelscited by 0
  11. 2026InsightEmb: Learning Action-Intent Embeddings for Agentic Insight Retrievalcited by 0