Deciphering Emotions in Children Storybooks: A Comparative Analysis of Multimodal LLMs in Educational Applications
On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding
Abstract (arXiv)
Emotion recognition capabilities in multimodal AI systems are crucial for developing culturally responsive educational technologies, yet remain underexplored for Arabic language contexts where culturally appropriate learning tools are critically needed. This study evaluates the emotion recognition performance of two advanced multimodal large language models, GPT-4o and Gemini 1.5 Pro, when processing Arabic children's storybook illustrations. We assessed both models across three prompting strategies (zero-shot, few-shot, and chain-of-thought) using 75 images from seven Arabic storybooks, comparing model predictions with human annotations based on Plutchik's emotional framework. GPT-4o consistently outperformed Gemini across all conditions, achieving the highest macro F1-score of 59% with chain-of-thought prompting compared to Gemini's best performance of 43%. Error analysis revealed systematic misclassification patterns, with valence inversions accounting for 60.7% of errors, while both models struggled with culturally nuanced emotions and ambiguous narrative contexts. These findings highlight fundamental limitations in current models' cultural understanding and emphasize the need for culturally sensitive training approaches to develop effective emotion-aware educational technologies for Arabic-speaking learners.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 10 of its 42 references that are in the map. First posted on arXiv 22 Jun 2025.
Builds on
10 of 42 references.
References · 10
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
- 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
- 2013Crowdsourcing a Word-Emotion Association Lexiconcited by 196
- 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 186
- 2020ARBERT & MARBERT: Deep Bidirectional Transformers for Arabiccited by 166
- 2021SpanEmo: Casting Multi-label Emotion Classification as Span-predictioncited by 37
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
- 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
- 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset
- 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
- 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
- 2026ArtECulture: Benchmarking Culture-Conditioned Visual Emotion Understanding in Multimodal Large Language Models
- 2025Pioneering Multimodal Emotion Recognition in the Era of Large Models: From Closed Sets to Open Vocabularies
- 2024Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding