mlsanityData to 7 Oct 2026

Paraphrasing Complex Network: Network Compression via Factor Transfer

arXiv 1802.04977 ↗First posted 14 Feb 2018cs.CVcited by 255

Not on the map. Why?

Abstract (arXiv)

Many researchers have sought ways of model compression to reduce the size of a deep neural network (DNN) with minimal performance degradation in order to use DNNs in embedded systems. Among the model compression methods, a method called knowledge transfer is to train a student network with a stronger teacher network. In this paper, we propose a novel knowledge transfer method which uses convolutional operations to paraphrase teacher's knowledge and to translate it for the student. This is done by two convolutional modules, which are called a paraphraser and a translator. The paraphraser is trained in an unsupervised manner to extract the teacher factors which are defined as paraphrased information of the teacher network. The translator located at the student network extracts the student factors and helps to translate the teacher factors by mimicking them. We observed that our student network trained with the proposed factor transfer method outperforms the ones trained with conventional knowledge transfer methods.

Key facts. Cited by 255 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 58; 2022: 54; 2023: 57; 2024: 58; 2025: 21; 2026: 7.

255 papers.

Sort:

By area:

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2022Decoupled Knowledge Distillationcited by 380
  3. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  4. 2022Knowledge Distillation from A Stronger Teachercited by 204
  5. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 165
  6. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  7. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  8. 2021Distilling Object Detectors via Decoupled Featurescited by 105
  9. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  10. 2023Deep Model Fusion: A Surveycited by 89
  11. 2022Learning Affordance Grounding from Exocentric Imagescited by 82
  12. 2021Show, Attend and Distill:Knowledge Distillation via Attention-based Feature Matchingcited by 76
38 more
  1. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 69
  2. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 64
  3. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learningcited by 61
  4. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillationcited by 58
  5. 2021Initialization and Regularization of Factorized Neural Layerscited by 58
  6. 2023Decoupled Kullback-Leibler Divergence Losscited by 58
  7. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  8. 2021A Survey on Green Deep Learningcited by 52
  9. 2021ReCU: Reviving the Dead Weights in Binary Neural Networkscited by 51
  10. 2023CrossKD: Cross-Head Knowledge Distillation for Object Detectioncited by 50
  11. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  12. 2021Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillationcited by 46
  13. 2021EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillationcited by 46
  14. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  15. 2021Complementary Relation Contrastive Distillationcited by 42
  16. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  17. 2022Cross-Architecture Knowledge Distillationcited by 40
  18. 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
  19. 2022Linkless Link Prediction via Relational Distillationcited by 39
  20. 2021Distilling Object Detectors with Feature Richnesscited by 38
  21. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  22. 2021Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Modelcited by 37
  23. 2021Learning Compatible Embeddingscited by 37
  24. 2022Masked Autoencoders Enable Efficient Knowledge Distillerscited by 36
  25. 2023Visual Tuningcited by 34
  26. 2021Learning from 2D: Contrastive Pixel-to-Point Knowledge Transfer for 3D Pretrainingcited by 32
  27. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  28. 2021Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translationcited by 31
  29. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  30. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30
  31. 2023DisWOT: Student Architecture Search for Distillation WithOut Trainingcited by 29
  32. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  33. 2021Self-slimmed Vision Transformercited by 27
  34. 2022Improved Feature Distillation via Projector Ensemblecited by 27
  35. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  36. 2021Multi-Scale Aligned Distillation for Low-Resolution Detectioncited by 24
  37. 2022Let Images Give You More:Point Cloud Cross-Modal Training for Shape Analysiscited by 24
  38. 2023TinyMIM: An Empirical Study of Distilling MIM Pre-trained Modelscited by 24

50 most cited of 255 All 255 (JSON)