mlsanityData to 7 Oct 2026

STICKERCONV: Generating Multimodal Empathetic Responses from Scratch

arXiv 2402.01679 ↗First posted 20 Jan 2024cs.CL cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Stickers, while widely recognized for enhancing empathetic communication in online interactions, remain underexplored in current empathetic dialogue research, notably due to the challenge of a lack of comprehensive datasets. In this paper, we introduce the Agent for STICKERCONV (Agent4SC), which uses collaborative agent interactions to realistically simulate human behavior with sticker usage, thereby enhancing multimodal empathetic communication. Building on this foundation, we develop a multimodal empathetic dialogue dataset, STICKERCONV, comprising 12.9K dialogue sessions, 5.8K unique stickers, and 2K diverse conversational scenarios. This dataset serves as a benchmark for multimodal empathetic generation. To advance further, we propose PErceive and Generate Stickers (PEGS), a multimodal empathetic response generation framework, complemented by a comprehensive set of empathy evaluation metrics based on LLM. Our experiments demonstrate PEGS's effectiveness in generating contextually relevant and emotionally resonant multimodal empathetic responses, contributing to the advancement of more nuanced and engaging empathetic dialogue systems.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 36 of its 69 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

36 of 69 references.

References · 36

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  12. 2016Improved Techniques for Training GANscited by 2,938
24 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  6. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  9. 2023Generating Images with Multimodal Language Modelscited by 259
  10. 2023Multimodal Large Language Models: A Surveycited by 193
  11. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  12. 2020MIME: MIMicking Emotions for Empathetic Response Generationcited by 114
  13. 2021CEM: Commonsense-aware Empathetic Response Generationcited by 89
  14. 2023Automated Annotation with Generative AI Requires Validationcited by 75
  15. 2023CharacterChat: Learning towards Conversational AI with Personalized Social Supportcited by 58
  16. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  17. 2020A Taxonomy of Empathetic Response Intents in Human Social Conversationscited by 51
  18. 2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43
  19. 2021CoMAE: A Multi-factor Hierarchical Framework for Empathetic Response Generationcited by 42
  20. 2020Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialogcited by 13
  21. 2023E-CORE: Emotion Correlation Enhanced Empathetic Dialogue Generationcited by 4
  22. 2022Reciprocity, Homophily, and Social Network Effects in Pictorial Communication: A Case Study of Bitmoji Stickerscited by 2
  23. 2023Should we trust web-scraped data?cited by 2
  24. 2022Selecting Stickers in Open-Domain Dialogue through Multitask Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 7.

21 papers.

Sort:

By area:

  1. 2025Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrievalcited by 26
  2. 2024Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspectivecited by 24
  3. 2025TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videoscited by 9
  4. 2024TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLMcited by 5
  5. 2025MGHFT: Multi-Granularity Hierarchical Fusion Transformer for Cross-Modal Sticker Emotion Recognitioncited by 4
  6. 2026A Multi-Agent Framework with Structured Reasoning and Reflective Refinement for Multimodal Empathetic Response Generationcited by 4
  7. 2025Enabling Chatbots with Eyes and Ears: An Immersive Multimodal Conversation System for Dynamic Interactionscited by 3
  8. 2024Impact of Stickers on Multimodal Sentiment and Intent in Social Media: A New Task, Dataset and Baselinecited by 2
  9. 2025When LLMs Team Up: The Emergence of Collaborative Affective Computingcited by 2
  10. 2025E3RG: Building Explicit Emotion-driven Empathetic Response Generation System with Multimodal Large Language Modelcited by 2
  11. 2025Emotion and Intention Guided Multi-Modal Learning for Sticker Response Selectioncited by 2
  12. 2024PerSRV: Personalized Sticker Retrieval with Vision-Language Modelcited by 1
9 more
  1. 2025Mixed Signals: Understanding Model Disagreement in Multimodal Empathy Detectioncited by 1
  2. 2026A Concise Agent is Less Expert: Revealing Side Effects of Using Style Features on Conversational Agentscited by 1
  3. 2025MemeCMD: An Automatically Generated Chinese Multi-turn Dialogue Dataset with Contextually Retrieved Memescited by 0
  4. 2025E-THER: A Multimodal Dataset for Empathic AI -- Towards Emotional Mismatch Awarenesscited by 0
  5. 2026Mechanistic Decoding of Cognitive Constructs in Large Language Modelscited by 0
  6. 2026PRIDE: Privileged Information-enhanced Distillation for Empathetic Dialogue Generationcited by 0
  7. 2026Towards High-Level Semantic Intelligencecited by 0
  8. 2026MOCC-R1: Reinforcing Reasoning-Response Consistency for Multimodal Counselor Response Generationcited by 0
  9. 2026EmAvatar: Multimodal Empathetic Response Generation via Conflict Resolution and Expressive Guidancecited by 0