# Accelerating Transfer Learning with Near-Data Computation on Cloud Object Stores

arXiv: https://arxiv.org/abs/2210.08650 · HTML: https://mlsanity.com/paper/2210.08650/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 115 references that are in the map.
- First posted on arXiv 16 Oct 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Transfusion: Understanding Transfer Learning for Medical Imaging](https://mlsanity.com/paper/1902.07208/) (2019)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Don't Decay the Learning Rate, Increase the Batch Size](https://mlsanity.com/paper/1711.00489/) (2017)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning](https://mlsanity.com/paper/2003.12862/) (2020)
- [Adversarial Multi-task Learning for Text Classification](https://mlsanity.com/paper/1704.05742/) (2017)
- [Glow: Graph Lowering Compiler Techniques for Neural Networks](https://mlsanity.com/paper/1805.00907/) (2018)
- [tf.data: A Machine Learning Data Processing Framework](https://mlsanity.com/paper/2101.12127/) (2021)
- [Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training](https://mlsanity.com/paper/2108.09373/) (2021)
- [Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce](https://mlsanity.com/paper/1711.08726/) (2017)
- [PaCa-ViT: Learning Patch-to-Cluster Attention in Vision Transformers](https://mlsanity.com/paper/2203.11987/) (2022)
- [Where Is My Training Bottleneck? Hidden Trade-Offs in Deep Learning Preprocessing Pipelines](https://mlsanity.com/paper/2202.08679/) (2022)
- [EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications](https://mlsanity.com/paper/2011.09463/) (2020)
- [An Empirical Guide to the Behavior and Use of Scalable Persistent Memory](https://mlsanity.com/paper/1908.03583/) (2019)
- [Effective Elastic Scaling of Deep Learning Workloads](https://mlsanity.com/paper/2006.13878/) (2020)
- [Effects of Layer Freezing on Transferring a Speech Recognition System to Under-resourced Languages](https://mlsanity.com/paper/2102.04097/) (2021)
- [Driving Experience Transfer Method for End-to-End Control of Self-Driving Cars](https://mlsanity.com/paper/1809.01822/) (2018)

## Abstract (arXiv)

Storage disaggregation underlies today's cloud and is naturally complemented by pushing down some computation to storage, thus mitigating the potential network bottleneck between the storage and compute tiers. We show how ML training benefits from storage pushdowns by focusing on transfer learning (TL), the widespread technique that democratizes ML by reusing existing knowledge on related tasks. We propose HAPI, a new TL processing system centered around two complementary techniques that address challenges introduced by disaggregation. First, applications must carefully balance execution across tiers for performance. HAPI judiciously splits the TL computation during the feature extraction phase yielding pushdowns that not only improve network time but also improve total TL training time by overlapping the execution of consecutive training iterations across tiers. Second, operators want resource efficiency from the storage-side computational resources. HAPI employs storage-side batch size adaptation allowing increased storage-side pushdown concurrency without affecting training accuracy. HAPI yields up to 2.5x training speed-up while choosing in 86.8% of cases the best performing split point or one that is at most 5% off from the best.