# Visual Prompting in LLMs for Enhancing Emotion Recognition

arXiv: https://arxiv.org/abs/2410.02244 · HTML: https://mlsanity.com/paper/2410.02244/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 3 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension](https://mlsanity.com/paper/2204.05991/) (2022)
- [Learning Emotion Representations from Verbal and Nonverbal Communication](https://mlsanity.com/paper/2305.13500/) (2023)
- [VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning](https://mlsanity.com/paper/2404.07078/) (2024)

## Built on by (most cited first, 12 of 12)

- [Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition](https://mlsanity.com/paper/2504.17224/) (2025)
- [VIBE: Can a VLM Read the Room?](https://mlsanity.com/paper/2506.11162/) (2025)
- [EmoCaliber: Advancing Reliable Visual Emotion Comprehension via Confidence Verbalization and Calibration](https://mlsanity.com/paper/2512.15528/) (2025)
- [ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting](https://mlsanity.com/paper/2603.23186/) (2026)
- [Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models](https://mlsanity.com/paper/2509.19595/) (2025)
- [VAEER: Visual Attention-Inspired Emotion Elicitation Reasoning](https://mlsanity.com/paper/2505.24342/) (2025)
- [Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?](https://mlsanity.com/paper/2512.17394/) (2025)
- [MemeCMD: An Automatically Generated Chinese Multi-turn Dialogue Dataset with Contextually Retrieved Memes](https://mlsanity.com/paper/2507.00891/) (2025)
- [Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations](https://mlsanity.com/paper/2603.22345/) (2026)
- [Facial-Expression-Aware Prompting for Empathetic LLM Tutoring](https://mlsanity.com/paper/2604.15336/) (2026)
- [XmoPipe: A Pipeline for Large-Scale In-the-Wild Human Motion Dataset Construction](https://mlsanity.com/paper/2606.20731/) (2026)
- [Decoding Affective Nuances: Enhancing MLLMs via Hierarchical Emotion Reasoning and Contrastive Discriminative Pruning](https://mlsanity.com/paper/2609.36782/) (2026)

## Abstract (arXiv)

Vision Large Language Models (VLLMs) are transforming the intersection of computer vision and natural language processing. Nonetheless, the potential of using visual prompts for emotion recognition in these models remains largely unexplored and untapped. Traditional methods in VLLMs struggle with spatial localization and often discard valuable global context. To address this problem, we propose a Set-of-Vision prompting (SoV) approach that enhances zero-shot emotion recognition by using spatial information, such as bounding boxes and facial landmarks, to mark targets precisely. SoV improves accuracy in face count and emotion categorization while preserving the enriched image context. Through a battery of experimentation and analysis of recent commercial or open-source VLLMs, we evaluate the SoV model's ability to comprehend facial expressions in natural environments. Our findings demonstrate the effectiveness of integrating spatial visual prompts into VLLMs for improving emotion recognition performance.