# Breaking the Scale Barrier: One-Shot Knowledge Transfer via Frequency Transform

arXiv: https://arxiv.org/abs/2603.07523 · HTML: https://mlsanity.com/paper/2603.07523/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 54 references that are in the map.
- First posted on arXiv 8 Mar 2026.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks](https://mlsanity.com/paper/1901.06523/) (2019)
- [Implicit Regularization in Matrix Factorization](https://mlsanity.com/paper/1705.09280/) (2017)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Learning to Grow Pretrained Models for Efficient Transformer Training](https://mlsanity.com/paper/2303.00980/) (2023)
- [Fine-Grained Car Detection for Visual Census Estimation](https://mlsanity.com/paper/1709.02480/) (2017)
- [The Herbarium Challenge 2019 Dataset](https://mlsanity.com/paper/1906.05372/) (2019)
- [Mimetic Initialization of Self-Attention Layers](https://mlsanity.com/paper/2305.09828/) (2023)
- [Neural Network Diffusion](https://mlsanity.com/paper/2402.13144/) (2024)
- [Towards Scalable and Versatile Weight Space Learning](https://mlsanity.com/paper/2406.09997/) (2024)
- [Initializing Models with Larger Ones](https://mlsanity.com/paper/2311.18823/) (2023)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [Learngene: From Open-World to Your Learning Task](https://mlsanity.com/paper/2106.06788/) (2021)
- [Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Models](https://mlsanity.com/paper/2305.02279/) (2023)
- [WAVE: Weight Templates for Adaptive Initialization of Variable-sized Models](https://mlsanity.com/paper/2406.17503/) (2024)
- [Harmonic Convolutional Networks based on Discrete Cosine Transform](https://mlsanity.com/paper/2001.06570/) (2020)
- [Transformer as Linear Expansion of Learngene](https://mlsanity.com/paper/2312.05614/) (2023)
- [Towards Effective Data-Free Knowledge Distillation via Diverse Diffusion Augmentation](https://mlsanity.com/paper/2410.17606/) (2024)
- [Jpeg Image Compression Using Discrete Cosine Transform - A Survey](https://mlsanity.com/paper/1405.6147/) (2014)

## Abstract (arXiv)

Transferring knowledge by fine-tuning large-scale pre-trained networks has become a standard paradigm for downstream tasks, yet the knowledge of a pre-trained model is tightly coupled with monolithic architecture, which restricts flexible reuse across models of varying scales. In response to this challenge, recent approaches typically resort to either parameter selection, which fails to capture the interdependent structure of this knowledge, or parameter prediction using generative models that depend on impractical access to large network collections. In this paper, we identify the low-frequency components of model weights as the concrete carrier of foundational, task-agnostic knowledge, its ``learngene", and validate this by demonstrating its efficient inheritance by downstream models and tasks. Based on this insight, we propose FRONT (FRequency dOmain kNowledge Transfer), a novel framework that uses the Discrete Cosine Transform (DCT) to isolate the low-frequency ``learngene". This learngene can be seamlessly adapted to initialize models of arbitrary size via simple truncation or padding, a process that is entirely training-free. For enhanced performance, we propose an optional low-cost refinement process that introduces a spectral regularizer to further improve the learngene's transferability. Extensive experiments demonstrate that FRONT achieves the state-of-the-art performance, accelerates convergence by up to 15× in vision tasks, and reduces training FLOPs by an average of 40.5% in language tasks. Code is available at https://github.com/LUcy0505/FRONT.