# Feature Fusion for Online Mutual Knowledge Distillation

arXiv: https://arxiv.org/abs/1904.09058 · HTML: https://mlsanity.com/paper/1904.09058/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2019.

## Built on by (most cited first, 30 of 40)

- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [A Good Student is Cooperative and Reliable: CNN-Transformer Collaborative Learning for Semantic Segmentation](https://mlsanity.com/paper/2307.12574/) (2023)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [Distilling a Powerful Student Model via Online Knowledge Distillation](https://mlsanity.com/paper/2103.14473/) (2021)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Revisiting Knowledge Distillation: An Inheritance and Exploration Framework](https://mlsanity.com/paper/2107.00181/) (2021)
- [Bi-Mapper: Holistic BEV Semantic Mapping for Autonomous Driving](https://mlsanity.com/paper/2305.04205/) (2023)
- [MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts](https://mlsanity.com/paper/2402.00893/) (2024)
- [HFT: Lifting Perspective Representations via Hybrid Feature Transformation](https://mlsanity.com/paper/2204.05068/) (2022)
- [QTI Submission to DCASE 2021: residual normalization for device-imbalanced acoustic scene classification with efficient design](https://mlsanity.com/paper/2206.13909/) (2022)
- [Not All Knowledge Is Created Equal: Mutual Distillation of Confident Knowledge](https://mlsanity.com/paper/2106.01489/) (2021)
- [Bootstrapping ViTs: Towards Liberating Vision Transformers from Pre-training](https://mlsanity.com/paper/2112.03552/) (2021)
- [Generalization Matters: Loss Minima Flattening via Parameter Hybridization for Efficient Online Knowledge Distillation](https://mlsanity.com/paper/2303.14666/) (2023)
- [Domain Generalization on Efficient Acoustic Scene Classification using Residual Normalization](https://mlsanity.com/paper/2111.06531/) (2021)
- [CAMERO: Consistency Regularized Ensemble of Perturbed Language Models with Weight Sharing](https://mlsanity.com/paper/2204.06625/) (2022)
- [Indirect Gradient Matching for Adversarial Robust Distillation](https://mlsanity.com/paper/2312.03286/) (2023)
- [CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation](https://mlsanity.com/paper/2408.13698/) (2024)
- [Collaborative Teacher-Student Learning via Multiple Knowledge Transfer](https://mlsanity.com/paper/2101.08471/) (2021)
- [Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning](https://mlsanity.com/paper/2407.02721/) (2024)
- [Prototype-based Personalized Pruning](https://mlsanity.com/paper/2103.15564/) (2021)
- [Efficient training of lightweight neural networks using Online Self-Acquired Knowledge Distillation](https://mlsanity.com/paper/2108.11798/) (2021)
- [Knowledge Distillation via Weighted Ensemble of Teaching Assistants](https://mlsanity.com/paper/2206.12005/) (2022)
- [Multi-View Fusion and Distillation for Subgrade Distresses Detection based on 3D-GPR](https://mlsanity.com/paper/2308.04779/) (2023)
- [Cooperative Meta-Learning with Gradient Augmentation](https://mlsanity.com/paper/2406.04639/) (2024)
- [Collaborative Knowledge Distillation via a Learning-by-Education Node Community](https://mlsanity.com/paper/2410.00074/) (2024)
- [Online Clustering of Seafloor Imagery for Interpretation during Long-Term AUV Operations](https://mlsanity.com/paper/2509.06678/) (2025)
- [Peer Collaborative Learning for Polyphonic Sound Event Detection](https://mlsanity.com/paper/2110.03511/) (2021)

## Abstract (arXiv)

We propose a learning framework named Feature Fusion Learning (FFL) that efficiently trains a powerful classifier through a fusion module which combines the feature maps generated from parallel neural networks. Specifically, we train a number of parallel neural networks as sub-networks, then we combine the feature maps from each sub-network using a fusion module to create a more meaningful feature map. The fused feature map is passed into the fused classifier for overall classification. Unlike existing feature fusion methods, in our framework, an ensemble of sub-network classifiers transfers its knowledge to the fused classifier and then the fused classifier delivers its knowledge back to each sub-network, mutually teaching one another in an online-knowledge distillation manner. This mutually teaching system not only improves the performance of the fused classifier but also obtains performance gain in each sub-network. Moreover, our model is more beneficial because different types of network can be used for each sub-network. We have performed a variety of experiments on multiple datasets such as CIFAR-10, CIFAR-100 and ImageNet and proved that our method is more effective than other alternative methods in terms of performance of both sub-networks and the fused classifier.