# TransTailor: Pruning the Pre-trained Model for Improved Transfer Learning

arXiv: https://arxiv.org/abs/2103.01542 · HTML: https://mlsanity.com/paper/2103.01542/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Explicit Inductive Bias for Transfer Learning with Convolutional Networks](https://mlsanity.com/paper/1802.01483/) (2018)
- [Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?](https://mlsanity.com/paper/1706.00712/) (2017)
- [Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1909.08174/) (2019)
- [Fast ConvNets Using Group-wise Brain Damage](https://mlsanity.com/paper/1506.02515/) (2015)
- [DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks](https://mlsanity.com/paper/1901.09229/) (2019)
- [Dynamic Slicing for Deep Neural Networks](https://mlsanity.com/paper/2009.13747/) (2020)

## Built on by (most cited first, 25 of 25)

- [Recent Advances on Federated Learning: A Systematic Survey](https://mlsanity.com/paper/2301.01299/) (2023)
- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs](https://mlsanity.com/paper/2312.15230/) (2023)
- [Interpretability of Machine Learning: Recent Advances and Future Prospects](https://mlsanity.com/paper/2305.00537/) (2023)
- [On the Generalization for Transfer Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2207.05377/) (2022)
- [Selectivity Drives Productivity: Efficient Dataset Pruning for Enhanced Transfer Learning](https://mlsanity.com/paper/2310.08782/) (2023)
- [Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation](https://mlsanity.com/paper/2208.12866/) (2022)
- [Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retraining](https://mlsanity.com/paper/2308.06755/) (2023)
- [Uncovering the Hidden Cost of Model Compression](https://mlsanity.com/paper/2308.14969/) (2023)
- [SCoTTi: Save Computation at Training Time with an adaptive framework](https://mlsanity.com/paper/2312.12483/) (2023)
- [Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptation](https://mlsanity.com/paper/2506.23675/) (2025)
- [DistFL: Distribution-aware Federated Learning for Mobile Scenarios](https://mlsanity.com/paper/2110.11619/) (2021)
- [Non-transferable Pruning](https://mlsanity.com/paper/2410.08015/) (2024)
- [Incremental Class Learning using Variational Autoencoders with Similarity Learning](https://mlsanity.com/paper/2110.01303/) (2021)
- [ODMTCNet: An Interpretable Multi-view Deep Neural Network Architecture for Image Feature Representation](https://mlsanity.com/paper/2110.14830/) (2021)
- [On The Relationship Between Continual Learning and Long-Tailed Recognition](https://mlsanity.com/paper/2306.13275/) (2023)
- [AC-Norm: Effective Tuning for Medical Image Analysis via Affine Collaborative Normalization](https://mlsanity.com/paper/2307.15282/) (2023)
- [Lightweight Change Detection in Heterogeneous Remote Sensing Images with Online All-Integer Pruning Training](https://mlsanity.com/paper/2405.01920/) (2024)
- [MO-EMT-NAS: Multi-Objective Continuous Transfer of Architectural Knowledge Between Tasks from Different Datasets](https://mlsanity.com/paper/2407.13122/) (2024)
- [Low-Complexity Inference in Continual Learning via Compressed Knowledge Transfer](https://mlsanity.com/paper/2505.08327/) (2025)
- [Neurosymbolic Artificial Intelligence for Robust Network Intrusion Detection: From Scratch to Transfer Learning](https://mlsanity.com/paper/2506.04454/) (2025)
- [Target Aware Network Architecture Search and Compression for Efficient Knowledge Transfer](https://mlsanity.com/paper/2205.05967/) (2022)
- [Neurosymbolic AI Transfer Learning Improves Network Intrusion Detection](https://mlsanity.com/paper/2509.10850/) (2025)
- [A Second-Order Perspective on Pruning at Initialization and Knowledge Transfer](https://mlsanity.com/paper/2509.24066/) (2025)
- [Carve VFM: Carving Specialists out of Generalists, Automatic Task-Specific Compression of Vision Foundation Models](https://mlsanity.com/paper/2610.16813/) (2026)

## Abstract (arXiv)

The increasing of pre-trained models has significantly facilitated the performance on limited data tasks with transfer learning. However, progress on transfer learning mainly focuses on optimizing the weights of pre-trained models, which ignores the structure mismatch between the model and the target task. This paper aims to improve the transfer performance from another angle - in addition to tuning the weights, we tune the structure of pre-trained models, in order to better match the target task. To this end, we propose TransTailor, targeting at pruning the pre-trained model for improved transfer learning. Different from traditional pruning pipelines, we prune and fine-tune the pre-trained model according to the target-aware weight importance, generating an optimal sub-model tailored for a specific target task. In this way, we transfer a more suitable sub-structure that can be applied during fine-tuning to benefit the final performance. Extensive experiments on multiple pre-trained models and datasets demonstrate that TransTailor outperforms the traditional pruning methods and achieves competitive or even better performance than other state-of-the-art transfer learning methods while using a smaller model. Notably, on the Stanford Dogs dataset, TransTailor can achieve 2.7% accuracy improvement over other transfer methods with 20% fewer FLOPs.