# Self-Training and Multi-Task Learning for Limited Data: Evaluation Study on Object Detection

arXiv: https://arxiv.org/abs/2309.06288 · HTML: https://mlsanity.com/paper/2309.06288/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Detection and dense prediction distillation, one of 12 directions in Knowledge distillation.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Placed in

- Knowledge distillation / [Detection and dense prediction distillation](https://mlsanity.com/distill/detection-and-dense-prediction-distillation/)

## Builds on (most cited first, 27 of 27)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Path Aggregation Network for Instance Segmentation](https://mlsanity.com/paper/1803.01534/) (2018)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Generalized Focal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection](https://mlsanity.com/paper/2006.04388/) (2020)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Libra R-CNN: Towards Balanced Learning for Object Detection](https://mlsanity.com/paper/1904.02701/) (2019)
- [Knowledge Distillation Meets Self-Supervision](https://mlsanity.com/paper/2006.07114/) (2020)
- [Data-Free Knowledge Distillation for Deep Neural Networks](https://mlsanity.com/paper/1710.07535/) (2017)
- [Data Distillation: Towards Omni-Supervised Learning](https://mlsanity.com/paper/1712.04440/) (2017)
- [Zero-Shot Knowledge Distillation in Deep Networks](https://mlsanity.com/paper/1905.08114/) (2019)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [Robust Learning Through Cross-Task Consistency](https://mlsanity.com/paper/2006.04096/) (2020)
- [Exploring Relational Context for Multi-Task Dense Prediction](https://mlsanity.com/paper/2104.13874/) (2021)
- [Localization Distillation for Dense Object Detection](https://mlsanity.com/paper/2102.12252/) (2021)
- [Knowledge Distillation for Multi-task Learning](https://mlsanity.com/paper/2007.06889/) (2020)
- [Automated Search for Resource-Efficient Branched Multi-Task Networks](https://mlsanity.com/paper/2008.10292/) (2020)
- [ThunderNet: Towards Real-time Generic Object Detection](https://mlsanity.com/paper/1903.11752/) (2019)
- [Learning Multiple Dense Prediction Tasks from Partially Annotated Data](https://mlsanity.com/paper/2111.14893/) (2021)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [ScratchDet: Training Single-Shot Object Detectors from Scratch](https://mlsanity.com/paper/1810.08425/) (2018)
- [Taskology: Utilizing Task Relations at Scale](https://mlsanity.com/paper/2005.07289/) (2020)
- [Partly Supervised Multitask Learning](https://mlsanity.com/paper/2005.02523/) (2020)
- [Localize to Classify and Classify to Localize: Mutual Guidance in Object Detection](https://mlsanity.com/paper/2009.14085/) (2020)
- [Knowledge Distillation for Object Detection: from generic to remote sensing datasets](https://mlsanity.com/paper/2307.09264/) (2023)

## Built on by (most cited first, 2 of 2)

- [Data exploitation: multi-task learning of object detection and semantic segmentation on partially annotated data](https://mlsanity.com/paper/2311.04040/) (2023)
- [Leveraging knowledge distillation for partial multi-task learning from multiple remote sensing datasets](https://mlsanity.com/paper/2405.15394/) (2024)

## Abstract (arXiv)

Self-training allows a network to learn from the predictions of a more complicated model, thus often requires well-trained teacher models and mixture of teacher-student data while multi-task learning jointly optimizes different targets to learn salient interrelationship and requires multi-task annotations for each training example. These frameworks, despite being particularly data demanding have potentials for data exploitation if such assumptions can be relaxed. In this paper, we compare self-training object detection under the deficiency of teacher training data where students are trained on unseen examples by the teacher, and multi-task learning with partially annotated data, i.e. single-task annotation per training example. Both scenarios have their own limitation but potentially helpful with limited annotated data. Experimental results show the improvement of performance when using a weak teacher with unseen data for training a multi-task student. Despite the limited setup we believe the experimental results show the potential of multi-task knowledge distillation and self-training, which could be beneficial for future study. Source code is at https://lhoangan.github.io/multas.