mlsanityData to 14 Oct 2026

Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models

arXiv 2510.24425 ↗First posted 28 Oct 2025cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

Recent efforts leverage knowledge distillation techniques to develop lightweight and practical sentiment analysis models. These methods are grounded in human-written instructions and large-scale user texts. Despite the promising results, two key challenges remain: (1) manually written instructions are limited in diversity and quantity, making them insufficient to ensure comprehensive coverage of distilled knowledge; (2) large-scale user texts incur high computational cost, hindering the practicality of these methods. To this end, we introduce CompEffDist, a comprehensive and efficient distillation framework for sentiment analysis. Our framework consists of two key modules: attribute-based automatic instruction construction and difficulty-based data filtering, which correspondingly tackle the aforementioned challenges. Applying our method across multiple model series (Llama-3, Qwen-3, and Gemma-3), we enable 3B student models to match the performance of 20x larger teacher models on most tasks. In addition, our approach greatly outperforms baseline methods in data efficiency, attaining the same performance level with only 10% of the data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 23 of its 38 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

23 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Pre-training Distillation for Large Language Models: A Design Space Exploration21 Oct
  2. 2024Revisiting Knowledge Distillation for Autoregressive Language Models19 Feb

Other references · 21

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2023LIMA: Less Is More for Alignmentcited by 1,100
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  6. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  8. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  9. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  10. 2022Is GPT-3 a Good Data Annotator?cited by 208
  11. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  12. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 109
9 more
  1. 2023Reasoning Implicit Sentiment with Chain-of-Thought Promptingcited by 80
  2. 2022How would Stance Detection Techniques Evolve after the Launch of ChatGPT?cited by 76
  3. 2024EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysiscited by 62
  4. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 55
  5. 2024Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimationcited by 53
  6. 2023MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Followingcited by 30
  7. 2021Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysiscited by 22
  8. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 12
  9. 2023InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPTcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CrisisKD: Five-Stage Knowledge Distillation for Aspect-Level Sentiment and Emotion Analysis in Crisis Discourse
  2. 2024Efficient Knowledge Distillation: Empowering Small Language Models with Teacher Model Insights
  3. 2023Domain-Expanded ASTE: Rethinking Generalization in Aspect Sentiment Triplet Extraction
  4. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models
  5. 2023Data-Free Distillation of Language Model by Text-to-Text Transfer
  6. 2024Linear Projections of Teacher Embeddings for Few-Class Distillation
  7. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
  8. 2024Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation