# Teaching with Commentaries

arXiv: https://arxiv.org/abs/2011.03037 · HTML: https://mlsanity.com/paper/2011.03037/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks](https://mlsanity.com/paper/2206.02916/) (2022)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [On Implicit Bias in Overparameterized Bilevel Optimization](https://mlsanity.com/paper/2212.14032/) (2022)
- [Meta-Learning to Improve Pre-Training](https://mlsanity.com/paper/2111.01754/) (2021)
- [Noether Networks: Meta-Learning Useful Conserved Quantities](https://mlsanity.com/paper/2112.03321/) (2021)
- [Learning to Scaffold: Optimizing Model Explanations for Teaching](https://mlsanity.com/paper/2204.10810/) (2022)
- [Auxiliary Learning as an Asymmetric Bargaining Game](https://mlsanity.com/paper/2301.13501/) (2023)
- [Complex Momentum for Optimization in Games](https://mlsanity.com/paper/2102.08431/) (2021)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [Adaptive Mixing of Auxiliary Losses in Supervised Learning](https://mlsanity.com/paper/2202.03250/) (2022)
- [When less is more: Simplifying inputs aids neural network understanding](https://mlsanity.com/paper/2201.05610/) (2022)
- [Curriculum Learning with Adam: The Devil Is in the Wrong Details](https://mlsanity.com/paper/2308.12202/) (2023)
- [Simple and Effective Gradient-Based Tuning of Sequence-to-Sequence Models](https://mlsanity.com/paper/2209.04683/) (2022)
- [Teacher Guided Training: An Efficient Framework for Knowledge Transfer](https://mlsanity.com/paper/2208.06825/) (2022)
- [Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search](https://mlsanity.com/paper/2104.00698/) (2021)

## Abstract (arXiv)

Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural network training examine teaching: providing learned information during the training process to improve downstream model performance. In this paper, we take steps towards extending the scope of teaching. We propose a flexible teaching framework using commentaries, learned meta-information helpful for training on a particular task. We present gradient-based methods to learn commentaries, leveraging recent work on implicit differentiation for scalability. We explore diverse applications of commentaries, from weighting training examples, to parameterising label-dependent data augmentation policies, to representing attention masks that highlight salient image regions. We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process. We also observe that commentaries generalise: they can be reused when training new models to obtain performance benefits, suggesting a use-case where commentaries are stored with a dataset and leveraged in future for improved model training.