mlsanityData to 14 Oct 2026

Local Brushstroke Quality Assessment via Vision-Language Feedback

arXiv 2607.16330 ↗First posted 15 Jul 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper investigates whether multimodal LLMs can evaluate local brushstroke quality in calligraphy and generate educationally useful natural language feedback. We construct an evaluation framework in which three multimodal LLMs (GPT-4o, Claude Sonnet 4, and Gemini 2.5 Flash) assess before-after image pairs of calligraphic works using a five-point ordinal scale, and compare their outputs against scores assigned by three expert calligraphers. We additionally examine a Retrieval-Augmented Generation (RAG) variant of Claude as a preliminary condition. Results show that all models achieve useful levels of absolute score accuracy (MAE), with GPT-4o performing best (MAE = 0.885). However, none of the models produce statistically significant overall rank correlations with human experts (Kendall's tau). Vocabulary analysis of generated rationales reveals characteristic evaluative biases in each model, and RAG is shown to improve rank correlation while worsening absolute accuracy, constituting an important negative result for text-based rule injection.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 16 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

7 of 16 references.

References · 7

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  6. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  7. 2025A Decade of Action Quality Assessment: Largest Systematic Survey of Trends, Challenges, and Future Directionscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HCSU: A Dataset and Benchmark for Fine-Grained Historical Calligraphy Style Understanding
  2. 2026Aesthetic Assessment of Chinese Handwritings Based on Vision Language Models
  3. 2025CalliReader: Contextualizing Chinese Calligraphy via an Embedding-Aligned Vision-Language Model
  4. 2026SketchJudge: A Diagnostic Benchmark for Grading Hand-drawn Diagrams with Multimodal Large Language Models
  5. 2026EduArt: An educational-level benchmark for evaluating art history knowledge in large language models
  6. 2024A Comprehensive Study of Multimodal Large Language Models for Image Quality Assessment
  7. 2026Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty?
  8. 2026How LLMs See Creativity: Zero-Shot Scoring of Visual Creativity with Interpretable Reasoning