# All Rivers Run to the Sea: Private Learning with Asymmetric Flows

arXiv: https://arxiv.org/abs/2312.05264 · HTML: https://mlsanity.com/paper/2312.05264/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 66 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Differential Privacy Has Disparate Impact on Model Accuracy](https://mlsanity.com/paper/1905.12101/) (2019)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Speeding up Convolutional Neural Networks with Low Rank Expansions](https://mlsanity.com/paper/1405.3866/) (2014)
- [The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks](https://mlsanity.com/paper/1911.07135/) (2019)
- [Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergences](https://mlsanity.com/paper/1807.01647/) (2018)
- [Membership Leakage in Label-Only Exposures](https://mlsanity.com/paper/2007.15528/) (2020)
- [Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware](https://mlsanity.com/paper/1806.03287/) (2018)
- [Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning](https://mlsanity.com/paper/2102.12677/) (2021)
- [Knowledge-Enriched Distributional Model Inversion Attacks](https://mlsanity.com/paper/2010.04092/) (2020)
- [Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attacks](https://mlsanity.com/paper/2201.12179/) (2022)
- [All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation](https://mlsanity.com/paper/1703.01827/) (2017)
- [DarkneTZ: Towards Model Privacy at the Edge using Trusted Execution Environments](https://mlsanity.com/paper/2004.05703/) (2020)
- [Privacy-Preserving Machine Learning with Fully Homomorphic Encryption for Deep Neural Network](https://mlsanity.com/paper/2106.07229/) (2021)
- [Label-Only Model Inversion Attacks via Boundary Repulsion](https://mlsanity.com/paper/2203.01925/) (2022)
- [SafetyNets: Verifiable Execution of Deep Neural Networks on an Untrusted Cloud](https://mlsanity.com/paper/1706.10268/) (2017)
- [Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clipping](https://mlsanity.com/paper/2009.03106/) (2020)
- [GAP: Differentially Private Graph Neural Networks with Aggregation Perturbation](https://mlsanity.com/paper/2203.00949/) (2022)
- [Advancements of federated learning towards privacy preservation: from federated learning to split learning](https://mlsanity.com/paper/2011.14818/) (2020)
- [MLCapsule: Guarded Offline Deployment of Machine Learning as a Service](https://mlsanity.com/paper/1808.00590/) (2018)
- [Parallel Multi Channel Convolution using General Matrix Multiplication](https://mlsanity.com/paper/1704.04428/) (2017)
- [Training CNNs with Low-Rank Filters for Efficient Image Classification](https://mlsanity.com/paper/1511.06744/) (2015)
- [On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks](https://mlsanity.com/paper/2011.05530/) (2020)

## Built on by (most cited first, 4 of 4)

- [Ethos: Rectifying Language Models in Orthogonal Parameter Space](https://mlsanity.com/paper/2403.08994/) (2024)
- [Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning](https://mlsanity.com/paper/2601.09076/) (2026)
- [Edge Private Graph Neural Networks with Singular Value Perturbation](https://mlsanity.com/paper/2403.10995/) (2024)
- [On the Relationship between Model Quantization and Model Inversion Attacks](https://mlsanity.com/paper/2610.00382/) (2026)

## Abstract (arXiv)

Data privacy is of great concern in cloud machine-learning service platforms, when sensitive data are exposed to service providers. While private computing environments (e.g., secure enclaves), and cryptographic approaches (e.g., homomorphic encryption) provide strong privacy protection, their computing performance still falls short compared to cloud GPUs. To achieve privacy protection with high computing performance, we propose Delta, a new private training and inference framework, with comparable model performance as non-private centralized training. Delta features two asymmetric data flows: the main information-sensitive flow and the residual flow. The main part flows into a small model while the residuals are offloaded to a large model. Specifically, Delta embeds the information-sensitive representations into a low-dimensional space while pushing the information-insensitive part into high-dimension residuals. To ensure privacy protection, the low-dimensional information-sensitive part is secured and fed to a small model in a private environment. On the other hand, the residual part is sent to fast cloud GPUs, and processed by a large model. To further enhance privacy and reduce the communication cost, Delta applies a random binary quantization technique along with a DP-based technique to the residuals before sharing them with the public platform. We theoretically show that Delta guarantees differential privacy in the public environment and greatly reduces the complexity in the private environment. We conduct empirical analyses on CIFAR-10, CIFAR-100 and ImageNet datasets and ResNet-18 and ResNet-34, showing that Delta achieves strong privacy protection, fast training, and inference without significantly compromising the model utility.