# DLink: Distilling Layer-wise and Dominant Knowledge from EEG Foundation Models

arXiv: https://arxiv.org/abs/2604.15016 · HTML: https://mlsanity.com/paper/2604.15016/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Distillation for medical imaging, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Placed in

- Knowledge distillation / [Distillation for medical imaging](https://mlsanity.com/distill/distillation-for-medical-imaging/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfaces](https://mlsanity.com/paper/1611.08024/) (2016)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI](https://mlsanity.com/paper/2405.18765/) (2024)
- [CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding](https://mlsanity.com/paper/2412.07236/) (2024)
- [DistiLLM: Towards Streamlined Distillation for Large Language Models](https://mlsanity.com/paper/2402.03898/) (2024)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [Less is More: Task-aware Layer-wise Distillation for Language Model Compression](https://mlsanity.com/paper/2210.01351/) (2022)
- [REVE: A Foundation Model for EEG -- Adapting to Any Setup with Large-Scale Pretraining on 25,000 Subjects](https://mlsanity.com/paper/2510.21585/) (2025)
- [Learning Student-Friendly Teacher Networks for Knowledge Distillation](https://mlsanity.com/paper/2102.07650/) (2021)
- [TSception: Capturing Temporal Dynamics and Spatial Asymmetry from EEG for Emotion Recognition](https://mlsanity.com/paper/2104.02935/) (2021)
- [Learning Optimal Representations with the Decodable Information Bottleneck](https://mlsanity.com/paper/2009.12789/) (2020)
- [CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Model](https://mlsanity.com/paper/2506.09110/) (2025)
- [FreeKD: Knowledge Distillation via Semantic Frequency Prompt](https://mlsanity.com/paper/2311.12079/) (2023)
- [EEG-Deformer: A Dense Convolutional Transformer for Brain-computer Interfaces](https://mlsanity.com/paper/2405.00719/) (2024)
- [EmT: A Novel Transformer for Generalized Cross-subject EEG Emotion Recognition](https://mlsanity.com/paper/2406.18345/) (2024)
- [Brain Foundation Models: A Survey on Advancements in Neural Signal Processing and Brain Discovery](https://mlsanity.com/paper/2503.00580/) (2025)
- [AdaBrain-Bench: Benchmarking Brain Foundation Models for Brain-Computer Interface Applications](https://mlsanity.com/paper/2507.09882/) (2025)
- [Are Large Brainwave Foundation Models Capable Yet? Insights from Fine-tuning](https://mlsanity.com/paper/2507.01196/) (2025)
- [How Convolutional Neural Networks Deal with Aliasing](https://mlsanity.com/paper/2102.07757/) (2021)
- [Geodesic Optimization for Predictive Shift Adaptation on EEG data](https://mlsanity.com/paper/2407.03878/) (2024)
- [Decoding Human Attentive States from Spatial-temporal EEG Patches Using Transformers](https://mlsanity.com/paper/2502.03736/) (2025)
- [Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models](https://mlsanity.com/paper/2512.01405/) (2025)
- [CMCRD: Cross-Modal Contrastive Representation Distillation for Emotion Recognition](https://mlsanity.com/paper/2504.09221/) (2025)

## Abstract (arXiv)

EEG foundation models (EFMs) achieve strong cross-subject and cross-task generalization through large-scale pretraining and downstream fine-tuning. Through empirical analysis, we observe that (i) task-adapted EFMs provide strong decoding performance but incur substantial overhead when retained as inference backbones, making knowledge distillation a natural route for optimizing compact students; and (ii) direct distillation from a fixed teacher representation underutilizes EFM knowledge, as task-discriminative information is distributed across intermediate layers rather than concentrated in the final layer. These observations motivate DLink (Distilling Layer-wise and Dominant Knowledge), a spectrally guided distillation framework with input-conditioned layer routing for transferring EFM knowledge into compact students. DLink uses a lightweight router to aggregate teacher layers for each input, and aligns magnitude and phase spectra to mitigate compression-induced spectral distortion in learned representations. The routed teacher knowledge is internalized by a project-then-compress student; the teacher and router are used only during training. Experiments on four EEG benchmarks show that DLink improves matched compact students and remains competitive with lightweight baselines, narrowing the gap to fine-tuned EFMs while substantially reducing parameters, FLOPs, and CPU-only inference latency.