# Efficient-Empathy: Towards Efficient and Effective Selection of Empathy Data

arXiv: https://arxiv.org/abs/2407.01937 · HTML: https://mlsanity.com/paper/2407.01937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 44 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 30)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024)
- [COMET: Commonsense Transformers for Automatic Knowledge Graph Construction](https://mlsanity.com/paper/1906.05317/) (2019)
- [MegaBlocks: Efficient Sparse Training with Mixture-of-Experts](https://mlsanity.com/paper/2211.15841/) (2022)
- [COSMIC: COmmonSense knowledge for eMotion Identification in Conversations](https://mlsanity.com/paper/2010.02795/) (2020)
- [Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM](https://mlsanity.com/paper/2403.07816/) (2024)
- [MoDS: Model-oriented Data Selection for Instruction Tuning](https://mlsanity.com/paper/2311.15653/) (2023)
- [MoEL: Mixture of Empathetic Listeners](https://mlsanity.com/paper/1908.07687/) (2019)
- [CEM: Commonsense-aware Empathetic Response Generation](https://mlsanity.com/paper/2109.05739/) (2021)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement](https://mlsanity.com/paper/2304.03946/) (2023)
- [Knowledge Bridging for Empathetic Dialogue Generation](https://mlsanity.com/paper/2009.09708/) (2020)
- [Harnessing the Power of Large Language Models for Empathetic Response Generation: Empirical Investigations and Improvements](https://mlsanity.com/paper/2310.05140/) (2023)
- [Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models](https://mlsanity.com/paper/2404.05567/) (2024)
- [Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training](https://mlsanity.com/paper/2405.03133/) (2024)
- [Rethinking the Instruction Quality: LIFT is What You Need](https://mlsanity.com/paper/2312.11508/) (2023)
- [Don't Lose Yourself! Empathetic Response Generation via Explicit Self-Other Awareness](https://mlsanity.com/paper/2210.03884/) (2022)
- [Think Twice: A Human-like Two-stage Conversational Agent for Emotional Response Generation](https://mlsanity.com/paper/2301.04907/) (2023)
- [Improving Empathetic Dialogue Generation by Dynamically Infusing Commonsense Knowledge](https://mlsanity.com/paper/2306.04657/) (2023)
- [Emp-RFT: Empathetic Response Generation via Recognizing Feature Transitions between Utterances](https://mlsanity.com/paper/2205.03112/) (2022)
- [From BERT to GPT-3 Codex: Harnessing the Potential of Very Large Language Models for Data Management](https://mlsanity.com/paper/2306.09339/) (2023)
- [Lingua Manga: A Generic Large Language Model Centric System for Data Curation](https://mlsanity.com/paper/2306.11702/) (2023)
- [An Iterative Associative Memory Model for Empathetic Response Generation](https://mlsanity.com/paper/2402.17959/) (2024)
- [Rational Sensibility: LLM Enhanced Empathetic Response Generation Guided by Self-presentation Theory](https://mlsanity.com/paper/2312.08702/) (2023)
- [Wish I Can Feel What You Feel: A Neural Approach for Empathetic Response Generation](https://mlsanity.com/paper/2212.02000/) (2022)
- [CTSM: Combining Trait and State Emotions for Empathetic Response Model](https://mlsanity.com/paper/2403.15516/) (2024)

## Abstract (arXiv)

In recent years, with the rapid advancements in large language models (LLMs), achieving excellent empathetic response capability has become a crucial prerequisite. Consequently, managing and understanding large-scale video datasets has gained increasing importance. However, empathetic data are typically trained without any quality selection, leading to inefficient data usage and wasted computational resources. Additionally, using raw data can result in low performance in empathetic dialogues. In this work, we present Efficient-Empathy, a sensibility and rationality score-based data selection algorithm that automatically selects sensibility and rationality data while discarding low-quality data. With only the sensibility data (59% of the full dataset), our trained sensibility model efficiently achieves state-of-the-art (SoTA) performance. Furthermore, with multiple data selection hyperparameters, the sensibility model demonstrates SoTA performance, showcasing the robustness of our method. By integrating sensibility and rationality data with a MoE structure, we achieve even higher performance, demonstrating the effectiveness of our Efficient-Empathy algorithm.