# Extraction of linearized models from pre-trained networks via knowledge distillation

arXiv: https://arxiv.org/abs/2604.06732 · HTML: https://mlsanity.com/paper/2604.06732/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 12 of its 35 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Photonics for artificial intelligence and neuromorphic computing](https://mlsanity.com/paper/2011.00111/) (2020)
- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [A Survey on Silicon Photonics for Deep Learning](https://mlsanity.com/paper/2101.01751/) (2021)
- [Extraction of nonlinearity in neural networks with Koopman operator](https://mlsanity.com/paper/2402.11740/) (2024)
- [Blending Optimal Control and Biologically Plausible Learning for Noise-Robust Physical Neural Networks](https://mlsanity.com/paper/2502.19053/) (2025)

## Abstract (arXiv)

Recent developments in hardware, such as photonic integrated circuits and optical devices, are driving demand for research on constructing machine learning architectures tailored for linear operations. Hence, it is valuable to explore methods for constructing learning machines with only linear operations after simple nonlinear preprocessing. In this study, we propose a framework to extract a linearized model from a pre-trained neural network for classification tasks by integrating Koopman operator theory with knowledge distillation. Numerical demonstrations on the MNIST and the Fashion-MNIST datasets reveal that the proposed model consistently outperforms the conventional least-squares-based Koopman approximation in both classification accuracy and numerical stability.