# Adaptive wavelet distillation from neural networks through interpretations

arXiv: https://arxiv.org/abs/2107.09145 · HTML: https://mlsanity.com/paper/2107.09145/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 67 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges](https://mlsanity.com/paper/2103.11251/) (2021)
- [Interpretable machine learning: definitions, methods, and applications](https://mlsanity.com/paper/1901.04592/) (2019)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Interpretations are useful: penalizing explanations to align neural networks with prior knowledge](https://mlsanity.com/paper/1909.13584/) (2019)
- [Beyond Word Importance: Contextual Decomposition to Extract Interactions from LSTMs](https://mlsanity.com/paper/1801.05453/) (2018)
- [Hierarchical interpretations for neural network predictions](https://mlsanity.com/paper/1806.05337/) (2018)
- [Multilevel Wavelet Decomposition Network for Interpretable Time Series Analysis](https://mlsanity.com/paper/1806.08946/) (2018)
- [WaveletKernelNet: An Interpretable Deep Neural Network for Industrial Intelligent Diagnosis](https://mlsanity.com/paper/1911.07925/) (2019)
- [Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2012.09413/) (2020)
- [Transformation Importance with Applications to Cosmology](https://mlsanity.com/paper/2003.01926/) (2020)
- [Learning Sparse Wavelet Representations](https://mlsanity.com/paper/1802.02961/) (2018)
- [Reducing the Teacher-Student Gap via Spherical Knowledge Disitllation](https://mlsanity.com/paper/2010.07485/) (2020)
- [Multiscale Adaptive Representation of Signals: I. The Basic Framework](https://mlsanity.com/paper/1507.04835/) (2015)

## Built on by (most cited first, 30 of 32)

- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Explaining black box text modules in natural language with language models](https://mlsanity.com/paper/2305.09863/) (2023)
- [Beyond Explaining: Opportunities and Challenges of XAI-Based Model Improvement](https://mlsanity.com/paper/2203.08008/) (2022)
- [Augmenting Interpretable Models with LLMs during Training](https://mlsanity.com/paper/2209.11799/) (2022)
- [Efficient Multi-scale Network with Learnable Discrete Wavelet Transform for Blind Motion Deblurring](https://mlsanity.com/paper/2401.00027/) (2023)
- [Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow](https://mlsanity.com/paper/2206.03198/) (2022)
- [Towards a theory of model distillation](https://mlsanity.com/paper/2403.09053/) (2024)
- [Fast Interpretable Greedy-Tree Sums](https://mlsanity.com/paper/2201.11931/) (2022)
- [Robust Time Series Denoising with Learnable Wavelet Packet Transform](https://mlsanity.com/paper/2206.06126/) (2022)
- [Scattering Spectra Models for Physics](https://mlsanity.com/paper/2306.17210/) (2023)
- [Efficiently Computing Sparse Fourier Transforms of q-ary Functions](https://mlsanity.com/paper/2301.06200/) (2023)
- [Machine Learning and Cosmology](https://mlsanity.com/paper/2203.08056/) (2022)
- [Learnable Wavelet Packet Transform for Data-Adapted Spectrograms](https://mlsanity.com/paper/2201.11069/) (2022)
- [Inferring dark matter substructure with astrometric lensing beyond the power spectrum](https://mlsanity.com/paper/2110.01620/) (2021)
- [YOLO-FEDER FusionNet: A Novel Deep Learning Architecture for Drone Detection](https://mlsanity.com/paper/2406.11641/) (2024)
- [Deep Polynomial Chaos Expansion](https://mlsanity.com/paper/2507.21273/) (2025)
- [One Wave To Explain Them All: A Unifying Perspective On Feature Attribution](https://mlsanity.com/paper/2410.01482/) (2024)
- [Interpretable Next-token Prediction via the Generalized Induction Head](https://mlsanity.com/paper/2411.00066/) (2024)
- [Assumption-lean falsification tests of rate double-robustness of double-machine-learning estimators](https://mlsanity.com/paper/2306.10590/) (2023)
- [Performance Optimization of YOLO-FEDER FusionNet for Robust Drone Detection in Visually Complex Environments](https://mlsanity.com/paper/2509.14012/) (2025)
- [Selecting Feature Interactions for Generalized Additive Models by Distilling Foundation Models](https://mlsanity.com/paper/2604.13332/) (2026)
- [Interpreting and improving deep-learning models with reality checks](https://mlsanity.com/paper/2108.06847/) (2021)
- [Spectral Regularization Allows Data-frugal Learning over Combinatorial Spaces](https://mlsanity.com/paper/2210.02604/) (2022)
- [Time Scale Network: A Shallow Neural Network For Time Series Data](https://mlsanity.com/paper/2311.06170/) (2023)
- [XAI-Enhanced Semantic Segmentation Models for Visual Quality Inspection](https://mlsanity.com/paper/2401.09900/) (2024)
- [WaveSim: A Wavelet-based Multi-scale Similarity Metric for Weather and Climate Fields](https://mlsanity.com/paper/2512.14656/) (2025)
- [Group Probability-Weighted Tree Sums for Interpretable Modeling of Heterogeneous Data](https://mlsanity.com/paper/2205.15135/) (2022)
- [Explaining Patterns in Data with Language Models via Interpretable Autoprompting](https://mlsanity.com/paper/2210.01848/) (2022)
- [WCCNet: Wavelet-context Cooperative Network for Efficient Multispectral Pedestrian Detection](https://mlsanity.com/paper/2308.01042/) (2023)
- [Effective Series Decomposition and Components Learning for Time Series Generation](https://mlsanity.com/paper/2511.00747/) (2025)

## Abstract (arXiv)

Recent deep-learning models have achieved impressive prediction performance, but often sacrifice interpretability and computational efficiency. Interpretability is crucial in many disciplines, such as science and medicine, where models must be carefully vetted or where interpretation is the goal itself. Moreover, interpretable models are concise and often yield computational efficiency. Here, we propose adaptive wavelet distillation (AWD), a method which aims to distill information from a trained neural network into a wavelet transform. Specifically, AWD penalizes feature attributions of a neural network in the wavelet domain to learn an effective multi-resolution wavelet transform. The resulting model is highly predictive, concise, computationally efficient, and has properties (such as a multi-scale structure) which make it easy to interpret. In close collaboration with domain experts, we showcase how AWD addresses challenges in two real-world settings: cosmological parameter inference and molecular-partner prediction. In both cases, AWD yields a scientifically interpretable and concise model which gives predictive performance better than state-of-the-art neural networks. Moreover, AWD identifies predictive features that are scientifically meaningful in the context of respective domains. All code and models are released in a full-fledged package available on Github (https://github.com/Yu-Group/adaptive-wavelets).