mlsanityData to 7 Oct 2026

Discrete Prompt Tuning via Recursive Utilization of Black-box Multimodal Large Language Model for Personalized Visual Emotion Recognition

arXiv 2509.04480 ↗First posted 30 Aug 2025cs.CL cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Multimodal emotion understanding

Abstract (arXiv)

Visual Emotion Recognition (VER) is an important research topic due to its wide range of applications, including opinion mining and advertisement design. Extending this capability to recognize emotions at the individual level further broadens its potential applications. Recently, Multimodal Large Language Models (MLLMs) have attracted increasing attention and demonstrated performance comparable to that of conventional VER methods. However, MLLMs are trained on large and diverse datasets containing general opinions, which causes them to favor majority viewpoints and familiar patterns. This tendency limits their performance in a personalized VER, which is crucial for practical and real-world applications, and indicates a key area for improvement. To address this limitation, the proposed method employs discrete prompt tuning inspired by the process of humans' prompt engineering to adapt the VER task to each individual. Our method selects the best natural language representation from the generated prompts and uses it to update the prompt for the realization of accurate personalized VER.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal emotion understanding, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 54 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

30 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024GPT as Psychologist? Preliminary Evaluations for GPT-4V on Visual Affective Computing9 Mar

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  7. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  8. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  9. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  10. 2023Large Language Models as General Pattern Machinescited by 196
  11. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  12. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
17 more
  1. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  2. 2016Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmarkcited by 71
  3. 2022Black-box Prompt Learning for Pre-trained Language Modelscited by 70
  4. 2015Robust Image Sentiment Analysis Using Progressively Trained and Domain Transferred Deep Networkscited by 46
  5. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  6. 2024EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuningcited by 42
  7. 2023GPT-4V with Emotion: A Zero-shot Benchmark for Generalized Emotion Recognitioncited by 38
  8. 2023Language Models as Black-Box Optimizers for Vision-Language Modelscited by 36
  9. 2023Learning Emotion Representations from Verbal and Nonverbal Communicationcited by 31
  10. 2022MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysiscited by 29
  11. 2022Affection: Learning Affective Explanations for Real-World Visual Datacited by 19
  12. 2019What Twitter Profile and Posted Images Reveal About Depression and Anxietycited by 17
  13. 2017Building Emotional Machines: Recognizing Image Emotions through Deep Neural Networkscited by 14
  14. 2024Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledgecited by 8
  15. 2023On the use of Vision-Language models for Visual Sentiment Analysis: a study on CLIPcited by 5
  16. 2024Disturbing Image Detection Using LMM-Elicited Emotion Embeddingscited by 4
  17. 2024A Survey of Deep Learning for Group-level Emotion Recognitioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models Meet Multimodal Emotion Recognition and Reasoning: A Survey
  2. 2025Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
  3. 2025Evaluating Vision-Language Models for Emotion Recognition
  4. 2025AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
  5. 2026MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
  6. 2025EmoVLM-KD: Fusing Distilled Expertise with Vision-Language Models for Visual Emotion Analysis
  7. 2026Visual Affect Analysis: Predicting Emotions of Image Viewers with Vision-Language Models
  8. 2025Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting