mlsanityData to 14 Oct 2026

Extraction of linearized models from pre-trained networks via knowledge distillation

arXiv 2604.06732 ↗First posted 8 Apr 2026cs.LGcited by 0

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Recent developments in hardware, such as photonic integrated circuits and optical devices, are driving demand for research on constructing machine learning architectures tailored for linear operations. Hence, it is valuable to explore methods for constructing learning machines with only linear operations after simple nonlinear preprocessing. In this study, we propose a framework to extract a linearized model from a pre-trained neural network for classification tasks by integrating Koopman operator theory with knowledge distillation. Numerical demonstrations on the MNIST and the Fashion-MNIST datasets reveal that the proposed model consistently outperforms the conventional least-squares-based Koopman approximation in both classification accuracy and numerical stability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 12 of its 35 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

12 of 35 references.

References · 12

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2021Modern Koopman Theory for Dynamical Systemscited by 276
  6. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 263
  7. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 202
  8. 2020Photonics for artificial intelligence and neuromorphic computingcited by 112
  9. 2021An optical neural network using less than 1 photon per multiplicationcited by 30
  10. 2021A Survey on Silicon Photonics for Deep Learningcited by 23
  11. 2024Extraction of nonlinearity in neural networks with Koopman operatorcited by 3
  12. 2025Blending Optimal Control and Biologically Plausible Learning for Noise-Robust Physical Neural Networkscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Towards Understanding Knowledge Distillation
  2. 2023Improving Knowledge Distillation with Teacher's Explanation
  3. 2023Knowledge Distillation Performs Partial Variance Reduction
  4. 2023RdimKD: Generic Distillation Paradigm by Dimensionality Reduction
  5. 2025Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling
  6. 2026Learnable Koopman-Enhanced Transformer-Based Time Series Forecasting with Spectral Control
  7. 2022Learning Invariant Subspaces of Koopman Operators--Part 1: A Methodology for Demonstrating a Dictionary's Approximate Subspace Invariance
  8. 2022Layerwise Bregman Representation Learning with Applications to Knowledge Distillation