# Small-Group Learning, with Application to Neural Architecture Search

arXiv: https://arxiv.org/abs/2012.12502 · HTML: https://mlsanity.com/paper/2012.12502/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2020.

## Built on by (most cited first, 2 of 2)

- [Learning by Examples Based on Multi-level Optimization](https://mlsanity.com/paper/2109.10824/) (2021)
- [Learning by Grouping: A Multilevel Optimization Framework for Improving Fairness in Classification without Losing Accuracy](https://mlsanity.com/paper/2304.00486/) (2023)

## Abstract (arXiv)

In human learning, an effective learning methodology is small-group learning: a small group of students work together towards the same learning objective, where they express their understanding of a topic to their peers, compare their ideas, and help each other to trouble-shoot problems. In this paper, we aim to investigate whether this human learning method can be borrowed to train better machine learning models, by developing a novel ML framework -- small-group learning (SGL). In our framework, a group of learners (ML models) with different model architectures collaboratively help each other to learn by leveraging their complementary advantages. Specifically, each learner uses its intermediately trained model to generate a pseudo-labeled dataset and re-trains its model using pseudo-labeled datasets generated by other learners. SGL is formulated as a multi-level optimization framework consisting of three learning stages: each learner trains a model independently and uses this model to perform pseudo-labeling; each learner trains another model using datasets pseudo-labeled by other learners; learners improve their architectures by minimizing validation losses. An efficient algorithm is developed to solve the multi-level optimization problem. We apply SGL for neural architecture search. Results on CIFAR-100, CIFAR-10, and ImageNet demonstrate the effectiveness of our method.