# Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling

arXiv: https://arxiv.org/abs/2609.01232 · HTML: https://mlsanity.com/paper/2609.01232/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Token pruning and merging for ViTs, one of 12 directions in Efficient LLM inference and serving.
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 1 Sep 2026.

## Placed in

- Efficient LLM inference and serving / [Token pruning and merging for ViTs](https://mlsanity.com/inference/token-pruning-and-merging-for-vits/)

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Token Merging: Your ViT But Faster](https://mlsanity.com/paper/2210.09461/) (2022)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Token Fusion: Bridging the Gap between Token Pruning and Token Merging](https://mlsanity.com/paper/2312.01026/) (2023)
- [SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learning](https://mlsanity.com/paper/2212.10986/) (2022)
- [A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking](https://mlsanity.com/paper/2309.02031/) (2023)
- [Permutation Equivariance of Transformers and Its Applications](https://mlsanity.com/paper/2304.07735/) (2023)
- [PRICURE: Privacy-Preserving Collaborative Inference in a Multi-Party Setting](https://mlsanity.com/paper/2102.09751/) (2021)
- [PATROL: Privacy-Oriented Pruning for Collaborative Inference Against Model Inversion Attacks](https://mlsanity.com/paper/2307.10981/) (2023)
- [Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge](https://mlsanity.com/paper/2509.09955/) (2025)
- [Make Split, not Hijack: Preventing Feature-Space Hijacking Attacks in Split Learning](https://mlsanity.com/paper/2404.09265/) (2024)
- [DRAG: Data Reconstruction Attack using Guided Diffusion](https://mlsanity.com/paper/2509.11724/) (2025)
- [From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognition](https://mlsanity.com/paper/2603.26336/) (2026)
- [On the (In-)Security of the Shuffling Defense in the Transformer Secure Inference](https://mlsanity.com/paper/2605.04901/) (2026)
- [Oops!... They Stole it Again: Attacks on Split Learning](https://mlsanity.com/paper/2508.10598/) (2025)

## Abstract (arXiv)

Vision Transformers (ViTs) are increasingly used in split-inference systems, where edge devices transmit intermediate token representations to a remote cloud. In this setting, token reduction lowers computation and communication costs, while token shuffling disrupts the spatial organization of the transmitted tokens, potentially limiting information leakage. However, their privacy benefits remain unclear against feature inversion attacks, which attempt to reconstruct the input from the transmitted embeddings. In this work, we show that, despite disrupting the spatial structure required by conventional reconstruction attacks, transmitted token embeddings retain substantial positional information. Based on this observation, we introduce the Spatially Aligned Reconstruction Attack (SARA), a unified pipeline that predicts token positions, restores their spatial layout, reconstructs missing embeddings using a feature-space masked autoencoder, and recovers the input image. Our results demonstrate that token shuffling provides only apparent privacy, as SARA largely reconstructs the original token organization. Token reduction offers stronger protection, but significant leakage persists when the retained tokens preserve sufficient semantic and positional information. Finally, we introduce a lightweight edge-side defense that removes positional embeddings and progressively adapts the edge-side transformer blocks through knowledge distillation. It substantially reduces attack performance against SARA, while preserving downstream task accuracy and requiring no changes to the cloud-side model.