# AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning

arXiv: https://arxiv.org/abs/2607.17913 · HTML: https://mlsanity.com/paper/2607.17913/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 20 Jul 2026.

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems](https://mlsanity.com/paper/1910.14315/) (2019)
- [Detailed comparison of communication efficiency of split learning and federated learning](https://mlsanity.com/paper/1909.09145/) (2019)
- [BottleNet: A Deep Learning Architecture for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1902.01000/) (2019)
- [SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2407.00952/) (2024)
- [Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data](https://mlsanity.com/paper/2206.08330/) (2022)
- [HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models](https://mlsanity.com/paper/2505.02795/) (2025)
- [BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing](https://mlsanity.com/paper/2201.02693/) (2022)
- [Reducing Communication for Split Learning by Randomized Top-k Sparsification](https://mlsanity.com/paper/2305.18469/) (2023)
- [Communication-Efficient Split Learning via Adaptive Feature-Wise Compression](https://mlsanity.com/paper/2307.10805/) (2023)
- [Lightweight compression of neural network feature tensors for collaborative intelligence](https://mlsanity.com/paper/2105.06002/) (2021)
- [C3-SL: Circular Convolution-Based Batch-Wise Compression for Communication-Efficient Split Learning](https://mlsanity.com/paper/2207.12397/) (2022)
- [Adaptive Compression-Aware Split Learning and Inference for Enhanced Network Efficiency](https://mlsanity.com/paper/2311.05739/) (2023)
- [Split Fine-Tuning for Large Language Models in Wireless Networks](https://mlsanity.com/paper/2501.09237/) (2025)
- [SplitLLM: Hierarchical Split Learning for Large Language Model over Wireless Network](https://mlsanity.com/paper/2501.13318/) (2025)
- [SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learning](https://mlsanity.com/paper/2412.17150/) (2024)
- [Energy-Efficient Split Learning for Fine-Tuning Large Language Models in Edge Networks](https://mlsanity.com/paper/2412.00090/) (2024)
- [SplitFrozen: Split Learning with Device-side Model Frozen for Fine-Tuning LLM on Heterogeneous Resource-Constrained Devices](https://mlsanity.com/paper/2503.18986/) (2025)
- [Communication Efficient Split Learning of ViTs with Attention-based Double Compression](https://mlsanity.com/paper/2509.15058/) (2025)
- [Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach](https://mlsanity.com/paper/2502.06355/) (2025)

## Built on by (most cited first, 1 of 1)

- [Contribution-Aware Bandwidth Allocation for Multimodal Split Learning](https://mlsanity.com/paper/2609.01406/) (2026)

## Abstract (arXiv)

Distributed Fine-Tuning (DFT) of large-scale Foundation Models (FMs) on resource-constrained edge devices is limited by local compute constraints and communication overhead. Parallel Split Learning (PSL) reduces client-side computation by keeping few model layers on each client and offloading the remaining computation to the server; however, clients must exchange intermediate activations and gradients with the server at every training step. Existing SL communication-compression methods mainly rely on task-agnostic heuristics, such as sparsification and quantization. While learnable SL compressors can better adapt to intermediate representations, they require co-training with the target model. Therefore, directly inserting them into off-the-shelf FMs introduces feature-distribution misalignment and degrades DFT performance. To address this, we propose AE-PSL, a communication-efficient PSL framework that compresses intermediate activations and gradients using a lightweight AutoEncoder (AE) placed at the split layer. To ensure compatibility of AE compression with pre-trained FMs, AE-PSL introduces a novel two-stage alignment mechanism, which adapts the AE to the pre-trained model's feature manifold and client-specific feature distributions before DFT.