# Two Independent Teachers are Better Role Model

arXiv: https://arxiv.org/abs/2306.05745 · HTML: https://mlsanity.com/paper/2306.05745/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 28 references that are in the map.
- First posted on arXiv 9 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Semi-Supervised Learning with Ladder Networks](https://mlsanity.com/paper/1507.02672/) (2015)
- [Shape-aware Semi-supervised 3D Semantic Segmentation for Medical Images](https://mlsanity.com/paper/2007.10732/) (2020)
- [Semi-Supervised Medical Image Segmentation via Learning Consistency under Transformations](https://mlsanity.com/paper/1911.01218/) (2019)
- [HyperDense-Net: A hyper-densely connected CNN for multi-modal image segmentation](https://mlsanity.com/paper/1804.02967/) (2018)
- [Semi-Supervised Brain Lesion Segmentation with an Adapted Mean Teacher Model](https://mlsanity.com/paper/1903.01248/) (2019)
- [Semi-Supervised Multi-Organ Segmentation via Deep Multi-Planar Co-Training](https://mlsanity.com/paper/1804.02586/) (2018)
- [Deep CNN ensembles and suggestive annotations for infant brain MRI segmentation](https://mlsanity.com/paper/1712.05319/) (2017)

## Built on by (most cited first, 1 of 1)

- [3D-EffiViTCaps: 3D Efficient Vision Transformer with Capsule for Medical Image Segmentation](https://mlsanity.com/paper/2403.16350/) (2024)

## Abstract (arXiv)

Recent deep learning models have attracted substantial attention in infant brain analysis. These models have performed state-of-the-art performance, such as semi-supervised techniques (e.g., Temporal Ensembling, mean teacher). However, these models depend on an encoder-decoder structure with stacked local operators to gather long-range information, and the local operators limit the efficiency and effectiveness. Besides, the MRI data contain different tissue properties (TPs) such as T1 and T2. One major limitation of these models is that they use both data as inputs to the segment process, i.e., the models are trained on the dataset once, and it requires much computational and memory requirements during inference. In this work, we address the above limitations by designing a new deep-learning model, called 3D-DenseUNet, which works as adaptable global aggregation blocks in down-sampling to solve the issue of spatial information loss. The self-attention module connects the down-sampling blocks to up-sampling blocks, and integrates the feature maps in three dimensions of spatial and channel, effectively improving the representation potential and discriminating ability of the model. Additionally, we propose a new method called Two Independent Teachers (2IT), that summarizes the model weights instead of label predictions. Each teacher model is trained on different types of brain data, T1 and T2, respectively. Then, a fuse model is added to improve test accuracy and enable training with fewer parameters and labels compared to the Temporal Ensembling method without modifying the network architecture. Empirical results demonstrate the effectiveness of the proposed method. The code is available at https://github.com/AfifaKhaled/Two-Independent-Teachers-are-Better-Role-Model.