# Assessing LLMs in Art Contexts: Critique Generation and Theory of Mind Evaluation

arXiv: https://arxiv.org/abs/2504.12805 · HTML: https://mlsanity.com/paper/2504.12805/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 42 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Builds on (most cited first, 14 of 14)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models](https://mlsanity.com/paper/2310.16755/) (2023)
- [People cannot distinguish GPT-4 from a human in a Turing test](https://mlsanity.com/paper/2405.08007/) (2024)
- [Large Language Models Pass the Turing Test](https://mlsanity.com/paper/2503.23674/) (2025)
- [Does GPT-4 pass the Turing test?](https://mlsanity.com/paper/2310.20216/) (2023)
- [Towards A Holistic Landscape of Situated Theory of Mind in Large Language Models](https://mlsanity.com/paper/2310.19619/) (2023)
- [GalleryGPT: Analyzing Paintings with Large Multimodal Models](https://mlsanity.com/paper/2408.00491/) (2024)
- [LLM Theory of Mind and Alignment: Opportunities and Risks](https://mlsanity.com/paper/2405.08154/) (2024)
- [ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter](https://mlsanity.com/paper/2305.07490/) (2023)
- [A Systematic Review on the Evaluation of Large Language Models in Theory of Mind Tasks](https://mlsanity.com/paper/2502.08796/) (2025)
- [GPT-4 is judged more human than humans in displaced and inverted Turing tests](https://mlsanity.com/paper/2407.08853/) (2024)

## Built on by (most cited first, 3 of 3)

- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [MIRAGE: A Micro-Interaction Relational Architecture for Grounded Exploration in Multi-Figure Artworks](https://mlsanity.com/paper/2604.23788/) (2026)
- [Towards High-Level Semantic Intelligence](https://mlsanity.com/paper/2607.24082/) (2026)

## Abstract (arXiv)

This study explored how large language models (LLMs) perform in two areas related to art: writing critiques of artworks and reasoning about mental states (Theory of Mind, or ToM) in art-related situations. For the critique generation part, we built a system that combines Noel Carroll's evaluative framework with a broad selection of art criticism theories. The model was prompted to first write a full-length critique and then shorter, more coherent versions using a step-by-step prompting process. These AI-generated critiques were then compared with those written by human experts in a Turing test-style evaluation. In many cases, human subjects had difficulty telling which was which, and the results suggest that LLMs can produce critiques that are not only plausible in style but also rich in interpretation, as long as they are carefully guided. In the second part, we introduced new simple ToM tasks based on situations involving interpretation, emotion, and moral tension, which can appear in the context of art. These go beyond standard false-belief tests and allow for more complex, socially embedded forms of reasoning. We tested 41 recent LLMs and found that their performance varied across tasks and models. In particular, tasks that involved affective or ambiguous situations tended to reveal clearer differences. Taken together, these results help clarify how LLMs respond to complex interpretative challenges, revealing both their cognitive limitations and potential. While our findings do not directly contradict the so-called Generative AI Paradox--the idea that LLMs can produce expert-like output without genuine understanding--they suggest that, depending on how LLMs are instructed, such as through carefully designed prompts, these models may begin to show behaviors that resemble understanding more closely than we might assume.