# Emotion-Aware Embedding Fusion in LLMs (Flan-T5, LLAMA 2, DeepSeek-R1, and ChatGPT 4) for Intelligent Response Generation

arXiv: https://arxiv.org/abs/2410.01306 · HTML: https://mlsanity.com/paper/2410.01306/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4](https://mlsanity.com/paper/2310.12321/) (2023)
- [Understanding the Benefits and Challenges of Using Large Language Model-based Conversational Agents for Mental Well-being Support](https://mlsanity.com/paper/2307.15810/) (2023)
- [BERT-ERC: Fine-tuning BERT is Enough for Emotion Recognition in Conversation](https://mlsanity.com/paper/2301.06745/) (2023)
- [Understanding Self-attention Mechanism via Dynamical System Perspective](https://mlsanity.com/paper/2308.09939/) (2023)

## Built on by (most cited first, 2 of 2)

- [Project Riley: Multimodal Multi-Agent LLM Collaboration with Emotional Reasoning and Voting](https://mlsanity.com/paper/2505.20521/) (2025)
- [Evaluating Human-LLM Representation Alignment: A Case Study on Affective Sentence Generation for Augmentative and Alternative Communication](https://mlsanity.com/paper/2503.11881/) (2025)

## Abstract (arXiv)

Empathetic and coherent responses are critical in auto-mated chatbot-facilitated psychotherapy. This study addresses the challenge of enhancing the emotional and contextual understanding of large language models (LLMs) in psychiatric applications. We introduce Emotion-Aware Embedding Fusion, a novel framework integrating hierarchical fusion and attention mechanisms to prioritize semantic and emotional features in therapy transcripts. Our approach combines multiple emotion lexicons, including NRC Emotion Lexicon, VADER, WordNet, and SentiWordNet, with state-of-the-art LLMs such as Flan-T5, LLAMA 2, DeepSeek-R1, and ChatGPT 4. Therapy session transcripts, comprising over 2,000 samples are segmented into hierarchical levels (word, sentence, and session) using neural networks, while hierarchical fusion combines these features with pooling techniques to refine emotional representations. Atten-tion mechanisms, including multi-head self-attention and cross-attention, further prioritize emotional and contextual features, enabling temporal modeling of emotion-al shifts across sessions. The processed embeddings, computed using BERT, GPT-3, and RoBERTa are stored in the Facebook AI similarity search vector database, which enables efficient similarity search and clustering across dense vector spaces. Upon user queries, relevant segments are retrieved and provided as context to LLMs, enhancing their ability to generate empathetic and con-textually relevant responses. The proposed framework is evaluated across multiple practical use cases to demonstrate real-world applicability, including AI-driven therapy chatbots. The system can be integrated into existing mental health platforms to generate personalized responses based on retrieved therapy session data.