mlsanityData to 7 Oct 2026

KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks

arXiv 2110.02432 ↗First posted 6 Oct 2021cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

We propose a new approach, Knowledge Distillation using Optimal Transport (KNOT), to distill the natural language semantic knowledge from multiple teacher networks to a student network. KNOT aims to train a (global) student model by learning to minimize the optimal transport cost of its assigned probability distribution over the labels to the weighted sum of probabilities predicted by the (local) teacher models, under the constraints, that the student model does not have access to teacher models' parameters or training data. To evaluate the quality of knowledge transfer, we introduce a new metric, Semantic Distance (SD), that measures semantic closeness between the predicted and ground truth label distributions. The proposed method shows improvements in the global model's SD performance over the baseline across three NLP tasks while performing on par with Entropy-based distillation on standard accuracy and F1 metrics. The implementation pertaining to this work is publicly available at: https://github.com/declare-lab/KNOT.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

30 of 47 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  6. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  8. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  9. 2020Knowledge Distillation: A Surveycited by 1,578
  10. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  11. 2019Contrastive Representation Distillationcited by 734
  12. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
18 more
  1. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  2. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  3. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  4. 2018Born Again Neural Networkscited by 539
  5. 2016Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filteringcited by 517
  6. 2019Agnostic Federated Learningcited by 451
  7. 2019Fair Resource Allocation in Federated Learningcited by 389
  8. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  9. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  10. 2015Learning with a Wasserstein Losscited by 194
  11. 2019Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advancescited by 118
  12. 2020Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Researchcited by 67
  13. 2018Differential Properties of Sinkhorn Approximation for Learning with Wasserstein Distancecited by 56
  14. 2018Algorithms and Theory for Multiple-Source Adaptationcited by 53
  15. 2019Knowledge Flow: Improve Upon Your Teacherscited by 36
  16. 2021Scaling Federated Learning for Fine-tuning of Large Language Modelscited by 36
  17. 2019FEED: Feature-level Ensemble for Knowledge Distillationcited by 23
  18. 2020Federated pretraining and fine tuning of BERT using clinical notes from multiple siloscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 1; 2026: 2.

8 papers.

  1. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  2. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  3. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  4. 2024Sinkhorn Distance Minimization for Knowledge Distillationcited by 15
  5. 2024Optimal Transport Guided Correlation Assignment for Multimodal Entity Linkingcited by 5
  6. 2025Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Datacited by 1
  7. 2026ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transportcited by 0
  8. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0