Projectable Models: One-Shot Generation of Small Specialized Transformers from Large Ones
Not on the map. Why?
Abstract (arXiv)
Modern Foundation Models (FMs) are typically trained on corpora spanning a wide range of different data modalities, topics and downstream tasks. Utilizing these models can be very computationally expensive and is out of reach for most consumer devices. Furthermore, most of the broad FM knowledge may actually be irrelevant for a specific task at hand. Here we explore a technique for mapping parameters of a large Transformer to parameters of a smaller specialized model. By making this transformation task-specific, we aim to capture a narrower scope of the knowledge needed for performing a specific task by a smaller model. We study our method on image modeling tasks, showing that performance of generated models exceeds that of universal conditional models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 6 Jun 2025.
Builds on
36 of 43 references.
References · 36
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2017Neural Discrete Representation Learningcited by 4,712
- 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,238
- 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
- 2016HyperNetworkscited by 1,027
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
- 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
- 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
- 2023Learning to Compress Prompts with Gist Tokenscited by 390
- 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
- 2019Continual learning with hypernetworkscited by 273
24 more
- 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
- 2018Graph HyperNetworks for Neural Architecture Searchcited by 132
- 2021Parameter Prediction for Unseen Deep Architecturescited by 89
- 2021HyperInverter: Improving StyleGAN Inversion via Hypernetworkcited by 85
- 2022HyperPrompt: Prompt-based Task-Conditioning of Transformerscited by 67
- 2020HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentationcited by 66
- 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
- 2019Deep Meta Functionals for Shape Representationcited by 53
- 2022HyperTuning: Toward Adapting Large Language Models without Back-propagationcited by 46
- 2024A Survey on Transformer Compressioncited by 46
- 2023Initializing Models with Larger Onescited by 39
- 2024Online Adaptation of Language Models with a Memory of Amortized Contextscited by 35
- 2024Streamlining Redundant Layers to Compress Large Language Modelscited by 35
- 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
- 2021Learning to Generate Task-Specific Adapters from Task Descriptioncited by 28
- 2022HINT: Hypernetwork Instruction Tuning for Efficient Zero- & Few-Shot Generalisationcited by 28
- 2024HyperMoE: Towards Better Mixture of Experts via Transferring Among Expertscited by 26
- 2023Learngene: Inheriting Condensed Knowledge from the Ancestry Model to Descendant Modelscited by 24
- 2020Weight Distillation: Transferring the Knowledge in Neural Network Parameterscited by 23
- 2022Hyperdecoders: Instance-specific decoders for multi-task NLPcited by 22
- 2022Boosting Natural Language Generation from Instructions with Meta-Learningcited by 15
- 2024MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learningcited by 15
- 2021HyperPocket: Generative Point Cloud Completioncited by 12
- 2024Investigating the Effectiveness of HyperTuning via Gistingcited by 4
Built on by
Not cited yet.