mlsanityData to 7 Oct 2026

Structured Knowledge Distillation for Dense Prediction

arXiv 1903.04197 ↗First posted 11 Mar 2019cs.CVcited by 277

Not on the map. Why?

Abstract (arXiv)

In this work, we consider transferring the structure information from large networks to compact ones for dense prediction tasks in computer vision. Previous knowledge distillation strategies used for dense prediction tasks often directly borrow the distillation scheme for image classification and perform knowledge distillation for each pixel separately, leading to sub-optimal performance. Here we propose to distill structured knowledge from large networks to compact networks, taking into account the fact that dense prediction is a structured prediction problem. Specifically, we study two structured distillation schemes: i) pair-wise distillation that distills the pair-wise similarities by building a static graph; and ii) holistic distillation that uses adversarial training to distill holistic knowledge. The effectiveness of our knowledge distillation approaches is demonstrated by experiments on three dense prediction tasks: semantic segmentation, depth estimation and object detection. Code is available at: https://git.io/StructKD

Key facts. Cited by 277 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Mar 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 53; 2022: 72; 2023: 55; 2024: 52; 2025: 23; 2026: 22.

277 papers.

Sort:

By area:

  1. 2024Depth Anything V2cited by 1,385
  2. 2021Learning Distilled Collaboration Graph for Multi-Agent Perceptioncited by 194
  3. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 140
  4. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  5. 2022Diffusion Probabilistic Model Made Slimcited by 121
  6. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 118
  7. 2022Cross-Image Relational Knowledge Distillation for Semantic Segmentationcited by 117
  8. 2022Curriculum Temperature for Knowledge Distillationcited by 110
  9. 2022Masked Generative Distillationcited by 107
  10. 2024Logit Standardization in Knowledge Distillationcited by 101
  11. 2021Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformerscited by 93
  12. 2021SimCVD: Simple Contrastive Voxel-Wise Representation Distillation for Semi-Supervised Medical Image Segmentationcited by 86
38 more
  1. 2023OccDepth: A Depth-Aware Method for 3D Semantic Scene Completioncited by 85
  2. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  3. 2022Knowledge Distillation via the Target-aware Transformercited by 70
  4. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillationcited by 69
  5. 2021LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detectorcited by 69
  6. 2021Robust Reference-based Super-Resolution via C2-Matchingcited by 68
  7. 2021Knowledge distillation from multi-modal to mono-modal segmentation networkscited by 68
  8. 2022MonoDistill: Learning Spatial Features for Monocular 3D Object Detectioncited by 68
  9. 2022DearKD: Data-Efficient Early Knowledge Distillation for Vision Transformerscited by 65
  10. 2022L2G: A Simple Local-to-Global Knowledge Transfer Framework for Weakly Supervised Semantic Segmentationcited by 57
  11. 2021Online Knowledge Distillation for Efficient Pose Estimationcited by 55
  12. 2023Knowledge Diffusion for Distillationcited by 51
  13. 2021EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillationcited by 46
  14. 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
  15. 2022Exploring Inter-Channel Correlation for Diversity-preserved KnowledgeDistillationcited by 44
  16. 2022The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillationcited by 44
  17. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  18. 2021Efficient Medical Image Segmentation Based on Knowledge Distillationcited by 43
  19. 2023UniDistill: A Universal Cross-Modality Knowledge Distillation Framework for 3D Object Detection in Bird's-Eye Viewcited by 43
  20. 2022Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillationcited by 41
  21. 2022Cross-Architecture Knowledge Distillationcited by 40
  22. 2023EdgeSAM: Prompt-In-the-Loop Distillation for SAMcited by 40
  23. 2022Wavelet Knowledge Distillation: Towards Efficient Image-to-Image Translationcited by 39
  24. 2023Knowledge Distillation on Graphs: A Surveycited by 38
  25. 2021Undistillable: Making A Nasty Teacher That CANNOT teach studentscited by 36
  26. 2024SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusioncited by 36
  27. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  28. 2023Structural and Statistical Texture Knowledge Distillation for Semantic Segmentationcited by 34
  29. 2023Visual Tuningcited by 34
  30. 2022PointDistiller: Structured Knowledge Distillation Towards Efficient and Compact 3D Detectioncited by 33
  31. 2023A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentationcited by 33
  32. 2023TinySAM: Pushing the Envelope for Efficient Segment Anything Modelcited by 33
  33. 2021Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detectioncited by 32
  34. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  35. 2021Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translationcited by 31
  36. 2021Data-Free Knowledge Transfer: A Surveycited by 31
  37. 2023Understanding the Role of the Projector in Knowledge Distillationcited by 31
  38. 2021Student Network Learning via Evolutionary Knowledge Distillationcited by 30

50 most cited of 277 All 277 (JSON)