mlsanityData to 7 Oct 2026

Cross-Image Relational Knowledge Distillation for Semantic Segmentation

arXiv 2204.06986 ↗First posted 14 Apr 2022cs.CVcited by 117

Not on the map. Why?

Abstract (arXiv)

Current Knowledge Distillation (KD) methods for semantic segmentation often guide the student to mimic the teacher's structured information generated from individual data samples. However, they ignore the global semantic relations among pixels across various images that are valuable for KD. This paper proposes a novel Cross-Image Relational KD (CIRKD), which focuses on transferring structured pixel-to-pixel and pixel-to-region relations among the whole images. The motivation is that a good teacher network could construct a well-structured feature space in terms of global pixel dependencies. CIRKD makes the student mimic better structured semantic relations from the teacher, thus improving the segmentation performance. Experimental results over Cityscapes, CamVid and Pascal VOC datasets demonstrate the effectiveness of our proposed approach against state-of-the-art distillation methods. The code is available at https://github.com/winycg/CIRKD.

Key facts. Cited by 117 papers in the map (data through 7 Oct 2026). Builds on 41 of its 51 references that are in the map. First posted on arXiv 14 Apr 2022.

Builds on

41 of 51 references.

References · 41

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  7. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  8. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  9. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  10. 2016Pyramid Scene Parsing Networkcited by 2,472
  11. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,154
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
29 more
  1. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  2. 2019Contrastive Multiview Codingcited by 1,297
  3. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  5. 2019Relational Knowledge Distillationcited by 873
  6. 2018BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentationcited by 645
  7. 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 573
  8. 2019Similarity-Preserving Knowledge Distillationcited by 507
  9. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
  10. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  11. 2016ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentationcited by 291
  12. 2019Correlation Congruence for Knowledge Distillationcited by 280
  13. 2019Structured Knowledge Distillation for Dense Predictioncited by 277
  14. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  15. 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
  16. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  17. 2020Channel-wise Knowledge Distillation for Dense Predictioncited by 161
  18. 2018ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentationcited by 133
  19. 2021SEED: Self-supervised Distillation For Visual Representationcited by 132
  20. 2019Cross-Batch Memory for Embedding Learningcited by 130
  21. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  22. 2019Knowledge Adaptation for Efficient Semantic Segmentationcited by 105
  23. 2020Efficient Semantic Video Segmentation with Per-frame Inferencecited by 63
  24. 2021Hierarchical Self-supervised Augmented Knowledge Distillationcited by 44
  25. 2021Mutual Contrastive Learning for Visual Representation Learningcited by 42
  26. 2018Improving Fast Segmentation With Teacher-student Learningcited by 34
  27. 2019Gated Convolutional Networks with Hybrid Connectivity for Image Classificationcited by 14
  28. 2020Multi-view Contrastive Learning for Online Knowledge Distillationcited by 11
  29. 2019Multi-Objective Pruning for CNNs Using Genetic Algorithmcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 28; 2024: 32; 2025: 29; 2026: 17.

117 papers.

Sort:

By area:

  1. 2022Knowledge Distillation from A Stronger Teachercited by 204
  2. 2023Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysiscited by 113
  3. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  4. 2024PromptKD: Unsupervised Prompt Distillation for Vision-Language Modelscited by 72
  5. 2025Frequency-Aligned Knowledge Distillation for Lightweight Spatiotemporal Forecastingcited by 53
  6. 2023Knowledge Diffusion for Distillationcited by 51
  7. 2023NORM: Knowledge Distillation via N-to-One Representation Matchingcited by 44
  8. 2022Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognitioncited by 36
  9. 2022ViTKD: Practical Guidelines for ViT feature knowledge distillationcited by 34
  10. 2023Visual Tuningcited by 34
  11. 2022MixSKD: Self-Knowledge Distillation from Mixup for Image Recognitioncited by 30
  12. 2023Directional Connectivity-based Segmentation of Medical Imagescited by 30
38 more
  1. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
  2. 2024Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Informationcited by 25
  3. 2024SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Modelscited by 24
  4. 2022Masked Distillation with Receptive Tokenscited by 23
  5. 2024DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecturecited by 21
  6. 2024OmniBind: Teach to Build Unequal-Scale Modality Interaction for Omni-Bind of Allcited by 20
  7. 2024MPQ-DM: Mixed Precision Quantization for Extremely Low Bit Diffusion Modelscited by 20
  8. 2023Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillationcited by 19
  9. 2025Merlin: Multi-View Representation Learning for Robust Multivariate Time Series Forecasting with Unfixed Missing Ratescited by 19
  10. 2025ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillationcited by 18
  11. 2023KD-DLGAN: Data Limited Image Generation via Knowledge Distillationcited by 16
  12. 2025Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognitioncited by 16
  13. 2022Accelerating Diffusion Sampling with Classifier-based Feature Distillationcited by 15
  14. 2023BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentationcited by 15
  15. 2024PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalizationcited by 15
  16. 2022FAKD: Feature Augmented Knowledge Distillation for Semantic Segmentationcited by 14
  17. 2023MaGIC: Multi-modality Guided Image Completioncited by 14
  18. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  19. 2024How to Train the Teacher Model for Effective Knowledge Distillationcited by 13
  20. 2022SERE: Exploring Feature Self-relation for Self-supervised Transformercited by 12
  21. 2022KD-DETR: Knowledge Distillation for Detection Transformer with Consistent Distillation Points Samplingcited by 12
  22. 2023Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labelscited by 12
  23. 2023NormKD: Normalized Logits for Knowledge Distillationcited by 12
  24. 2025MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillationcited by 12
  25. 2023Cloud-Device Collaborative Learning for Multimodal Large Language Modelscited by 11
  26. 2025MMT-ARD: Multimodal Multi-Teacher Adversarial Distillation for Robust Vision-Language Modelscited by 11
  27. 2026Distilling Time Series Foundation Models for Efficient Forecastingcited by 11
  28. 2022Weak-shot Semantic Segmentation via Dual Similarity Transfercited by 10
  29. 2023Deep Transfer Learning for Intelligent Vehicle Perception: a Surveycited by 10
  30. 2026ReCo-KD: Region- and Context-Aware Knowledge Distillation for Efficient 3D Medical Image Segmentationcited by 10
  31. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  32. 2024Knowledge Distillation with Refined Logitscited by 9
  33. 2025Structural and Statistical Texture Knowledge Distillation and Learning for Segmentationcited by 9
  34. 2025SRKD: Towards Efficient 3D Point Cloud Segmentation via Structure- and Relation-aware Knowledge Distillationcited by 9
  35. 2023Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentationcited by 8
  36. 2025Semi-supervised Semantic Segmentation with Multi-Constraint Consistency Learningcited by 8
  37. 2025Enhancing Image Generation Fidelity via Progressive Promptscited by 7
  38. 2024CKD: Contrastive Knowledge Distillation from A Sample-wise Perspectivecited by 6

50 most cited of 117 All 117 (JSON)