# On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference

arXiv: https://arxiv.org/abs/2605.04901 · HTML: https://mlsanity.com/paper/2605.04901/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 6 May 2026.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [High Accuracy and High Fidelity Extraction of Neural Networks](https://mlsanity.com/paper/1909.01838/) (2019)
- [Stealing Part of a Production Language Model](https://mlsanity.com/paper/2403.06634/) (2024)
- [CrypTen: Secure Multi-Party Computation Meets Machine Learning](https://mlsanity.com/paper/2109.00984/) (2021)
- [Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption](https://mlsanity.com/paper/2407.18003/) (2024)
- [CrypTFlow2: Practical 2-Party Secure Inference](https://mlsanity.com/paper/2010.06457/) (2020)
- [MPCFormer: fast, performant and private Transformer inference with MPC](https://mlsanity.com/paper/2211.01452/) (2022)
- [Cryptanalytic Extraction of Neural Network Models](https://mlsanity.com/paper/2003.04884/) (2020)
- [THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption](https://mlsanity.com/paper/2206.00216/) (2022)
- [PUMA: Secure Inference of LLaMA-7B in Five Minutes](https://mlsanity.com/paper/2307.12533/) (2023)
- [Selective Network Linearization for Efficient Private Inference](https://mlsanity.com/paper/2202.02340/) (2022)
- [SIRNN: A Math Library for Secure RNN Inference](https://mlsanity.com/paper/2105.04236/) (2021)
- [MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attention](https://mlsanity.com/paper/2211.13955/) (2022)
- [Nimbus: Secure and Efficient Two-Party Inference for Transformers](https://mlsanity.com/paper/2411.15707/) (2024)
- [PermLLM: Private Inference of Large Language Models within 3 Seconds under WAN](https://mlsanity.com/paper/2405.18744/) (2024)
- [CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference](https://mlsanity.com/paper/2412.10652/) (2024)
- [BAYHENN: Combining Bayesian Deep Learning and Homomorphic Encryption for Secure DNN Inference](https://mlsanity.com/paper/1906.00639/) (2019)
- [Towards Secure and Practical Machine Learning via Secret Sharing and Random Permutation](https://mlsanity.com/paper/2108.07463/) (2021)

## Built on by (most cited first, 2 of 2)

- [Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling](https://mlsanity.com/paper/2609.01232/) (2026)
- [A Graph-Based Analysis of Comparability, Scale, and Attribution in Private Transformer Inference](https://mlsanity.com/paper/2610.19607/) (2026)

## Abstract (arXiv)

For Transformer models, cryptographically secure inference ensures that the client learns only the final output, while the server learns nothing about the client's input. However, securely computing nonlinear layers remains a major efficiency bottleneck due to the substantial communication rounds and data transmission required. To address this issue, prior works reveal intermediate activations to the client, allowing nonlinear operations to be computed in plaintext. Although this approach significantly improves efficiency, exposing activations enables adversaries to extract model weights. To mitigate this risk, existing works employ a shuffling defense that reveals only randomly permuted activations to the client. In this work, we show that the shuffling defense is not as robust as previously claimed. We propose an attack that aligns differently shuffled activations to a common permutation and subsequently exploits them to extract model weights. Experiments on Pythia-70m and GPT-2 demonstrate that the proposed attack can align shuffled activations with mean squared errors ranging from 10-9 to 10-6. With a query cost of approximately 1, the adversary can recover model weights with L1-norm differences ranging from 10-4 to 10-2 compared to the oracle weights.