# VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoning

arXiv: https://arxiv.org/abs/2404.07078 · HTML: https://mlsanity.com/paper/2404.07078/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 32 of its 66 references that are in the map.
- First posted on arXiv 10 Apr 2024.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [KAT: A Knowledge Augmented Transformer for Vision-and-Language](https://mlsanity.com/paper/2112.08614/) (2021)
- [Context-Aware Emotion Recognition Networks](https://mlsanity.com/paper/1908.05913/) (2019)
- [ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System](https://mlsanity.com/paper/2304.14407/) (2023)
- [REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2206.01201/) (2022)
- [EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege's Principle](https://mlsanity.com/paper/2003.06692/) (2020)
- [Prompting Visual-Language Models for Dynamic Facial Expression Recognition](https://mlsanity.com/paper/2308.13382/) (2023)
- [Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions](https://mlsanity.com/paper/2304.04227/) (2023)
- [STEP: Spatial Temporal Graph Convolutional Networks for Emotion Perception from Gaits](https://mlsanity.com/paper/1910.12906/) (2019)
- [ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild](https://mlsanity.com/paper/1808.09568/) (2018)
- [Context De-confounded Emotion Recognition](https://mlsanity.com/paper/2303.11921/) (2023)
- [Learning Emotion Representations from Verbal and Nonverbal Communication](https://mlsanity.com/paper/2305.13500/) (2023)
- [EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition](https://mlsanity.com/paper/2310.16640/) (2023)
- [DialogueLLM: Context and Emotion Knowledge-Tuned Large Language Models for Emotion Recognition in Conversations](https://mlsanity.com/paper/2310.11374/) (2023)
- [CLIPER: A Unified Vision-Language Framework for In-the-Wild Facial Expression Recognition](https://mlsanity.com/paper/2303.00193/) (2023)
- [Robust Emotion Recognition in Context Debiasing](https://mlsanity.com/paper/2403.05963/) (2024)
- [Emotion Understanding in Videos Through Body, Context, and Visual-Semantic Embedding Loss](https://mlsanity.com/paper/2010.16396/) (2020)
- [A Simple Baseline for Knowledge-Based Visual Question Answering](https://mlsanity.com/paper/2310.13570/) (2023)

## Built on by (most cited first, 16 of 16)

- [Evaluating Vision-Language Models for Emotion Recognition](https://mlsanity.com/paper/2502.05660/) (2025)
- [Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer](https://mlsanity.com/paper/2405.19100/) (2024)
- [MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understanding](https://mlsanity.com/paper/2502.03183/) (2025)
- [ExpLLM: Towards Chain of Thought for Facial Expression Recognition](https://mlsanity.com/paper/2409.02828/) (2024)
- [Large Vision-Language Models as Emotion Recognizers in Context Awareness](https://mlsanity.com/paper/2407.11300/) (2024)
- [Visual Prompting in LLMs for Enhancing Emotion Recognition](https://mlsanity.com/paper/2410.02244/) (2024)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Visual and Textual Prompts in VLLMs for Enhancing Emotion Recognition](https://mlsanity.com/paper/2504.17224/) (2025)
- [CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs](https://mlsanity.com/paper/2410.01696/) (2024)
- [A Hitchhikers Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning](https://mlsanity.com/paper/2410.00485/) (2024)
- [Why We Feel: Breaking Boundaries in Emotional Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2504.07521/) (2025)
- [Anatomy of a Feeling: Narrating Embodied Emotions via Large Vision-Language Models](https://mlsanity.com/paper/2509.19595/) (2025)
- [Lotus at SemEval-2025 Task 11: RoBERTa with Llama-3 Generated Explanations for Multi-Label Emotion Classification](https://mlsanity.com/paper/2502.19935/) (2025)
- [EmoGist: Efficient In-Context Learning for Visual Emotion Understanding](https://mlsanity.com/paper/2505.14660/) (2025)
- [Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation](https://mlsanity.com/paper/2604.25255/) (2026)
- [Context-Aware Mixture of Domain Experts for Bodily Expression of Emotion in the Wild](https://mlsanity.com/paper/2608.02331/) (2026)

## Abstract (arXiv)

Recognising emotions in context involves identifying an individual's apparent emotions while considering contextual cues from the surrounding scene. Previous approaches to this task have typically designed explicit scene-encoding architectures or incorporated external scene-related information, such as captions. However, these methods often utilise limited contextual information or rely on intricate training pipelines to decouple noise from relevant information. In this work, we leverage the capabilities of Vision-and-Large-Language Models (VLLMs) to enhance in-context emotion classification in a more straightforward manner. Our proposed method follows a simple yet effective two-stage approach. First, we prompt VLLMs to generate natural language descriptions of the subject's apparent emotion in relation to the visual context. Second, the descriptions, along with the visual input, are used to train a transformer-based architecture that fuses text and visual features before the final classification task. This method not only simplifies the training process but also significantly improves performance. Experimental results demonstrate that the textual descriptions effectively guide the model to constrain the noisy visual input, allowing our fused architecture to outperform individual modalities. Our approach achieves state-of-the-art performance across three datasets, BoLD, EMOTIC, and CAER-S, without bells and whistles. The code will be made publicly available on github: https://github.com/NickyFot/EmoCommonSense.git