mlsanityData to 7 Oct 2026

MPCFormer: fast, performant and private Transformer inference with MPC

arXiv 2211.01452 ↗First posted 2 Nov 2022cs.LG cs.CRcited by 67

Not on the map. Why?

Abstract (arXiv)

Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inference latency by more than 60x or significantly compromise the inference quality. In this paper, we design the framework MPCFORMER as a practical solution, using Secure Multi-Party Computation (MPC) and Knowledge Distillation (KD). Through extensive evaluations, we show that MPCFORMER significantly speeds up Transformer inference in MPC settings while achieving similar ML performance to the input model. On the IMDb dataset, it achieves similar performance to BERTBASE, while being 5.3x faster. On the GLUE benchmark, it achieves 97% performance of BERTBASE with a 2.2x speedup. MPCFORMER remains effective with different trained Transformer weights such as ROBERTABASE and larger models including BERTLarge. Code is available at https://github.com/MccRee177/MPCFormer.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 2 Nov 2022.

Builds on

30 of 51 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
18 more
  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  8. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  10. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2014Deeply-Supervised Netscited by 364
  13. 2018Gazelle: A Low Latency Framework for Secure Neural Network Inferencecited by 193
  14. 2021CrypTen: Secure Multi-Party Computation Meets Machine Learningcited by 131
  15. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  16. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  17. 2018Chameleon: A Hybrid Secure Computation Framework for Machine Learning Applicationscited by 67
  18. 2018Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inferencecited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 16; 2024: 23; 2025: 12; 2026: 15.

67 papers.

Sort:

By area:

  1. 2023Fast Distributed Inference Serving for Large Language Modelscited by 143
  2. 2023Offsite-Tuning: Transfer Learning without Full Modelcited by 77
  3. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  4. 2023PrivLM-Bench: A Multi-level Privacy Evaluation Benchmark for Language Modelscited by 35
  5. 2024PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restorationcited by 27
  6. 2023Cuttlefish: Low-Rank Model Training without All the Tuningcited by 25
  7. 2023Grounding Foundation Models through Federated Transfer Learning: A General Frameworkcited by 24
  8. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  9. 2023LLMs Can Understand Encrypted Prompt: Towards Privacy-Computing Friendly Transformerscited by 20
  10. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  11. 2024Encryption-Friendly LLM Architecturecited by 15
  12. 2024Nimbus: Secure and Efficient Two-Party Inference for Transformerscited by 15
38 more
  1. 2024No Free Lunch Theorem for Privacy-Preserving LLM Inferencecited by 13
  2. 2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
  3. 2023Does compressing activations help model parallel training?cited by 11
  4. 2023East: Efficient and Accurate Secure Transformer Framework for Inferencecited by 9
  5. 2025CipherPrune: Efficient and Scalable Private Transformer Inferencecited by 9
  6. 2023MERGE: Fast Private Text Generationcited by 8
  7. 2024SecFormer: Fast and Accurate Privacy-Preserving Inference for Transformer Models via SMPCcited by 7
  8. 2024Ditto: Quantization-aware Secure Inference of Transformers upon MPCcited by 7
  9. 2024PrivCirNet: Efficient Private Inference via Block Circulant Transformationcited by 7
  10. 2024Comet: A Communication-efficient and Performant Approximation for Private Transformer Inferencecited by 7
  11. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  12. 2024CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inferencecited by 6
  13. 2025PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databasescited by 6
  14. 2024MPC-Minimized Secure LLM Inferencecited by 5
  15. 2026Token Economics for LLM Agents: A Dual-View Study from Computing and Economicscited by 5
  16. 2023PriViT: Vision Transformers for Fast Private Inferencecited by 4
  17. 2023Optimized Layerwise Approximation for Efficient Private Inference on Fully Homomorphic Encryptioncited by 4
  18. 2024ObfuscaTune: Obfuscated Offsite Fine-tuning and Inference of Proprietary LLMs on Private Datasetscited by 4
  19. 2024CipherDM: Secure Three-Party Inference for Diffusion Model Samplingcited by 3
  20. 2024AERO: Entropy-Guided Framework for Private LLM Inferencecited by 3
  21. 2025Comet: Accelerating Private Inference for Large Language Model by Predicting Activation Sparsitycited by 3
  22. 2023Approximating ReLU on a Reduced Ring for Efficient MPC-based Private Inferencecited by 2
  23. 2024Taylor Unswift: Secured Weight Release for Large Language Models via Taylor Expansioncited by 2
  24. 2026Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)cited by 2
  25. 2023Compact: Approximating Complex Activation Functions for Secure Computationcited by 1
  26. 2023SelectFormer: Private and Practical Data Selection for Transformerscited by 1
  27. 2024A Framework for Cost-Effective and Self-Adaptive LLM Shaking and Recovery Mechanismcited by 1
  28. 2024Confidential Prompting: Privacy-preserving LLM Inference on Cloudcited by 1
  29. 2024PrivQuant: Communication-Efficient Private Inference with Quantized Network/Protocol Co-Optimizationcited by 1
  30. 2024TruncFormer: Private LLM Inference Using Only Truncationscited by 1
  31. 2025A Framework for Double-Blind Federated Adaptation of Foundation Modelscited by 1
  32. 2025Private Transformer Inference in MLaaS: A Surveycited by 1
  33. 2025SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPCcited by 1
  34. 2025ENSI: Efficient Non-Interactive Secure Inference for Large Language Modelscited by 1
  35. 2026CGF-Softmax: A Cumulant-Based Softmax Reformulation for Efficient Inference under Homomorphic Encryptioncited by 1
  36. 2026P2RAG: Efficient Privacy-Preserving RAG Service Supporting Arbitrary Top-k Retrievalcited by 1
  37. 2026SecureRouter: Encrypted Routing for Efficient Secure Inferencecited by 1
  38. 2026Privacy-Preserving LLMs Routingcited by 1

50 most cited of 67 All 67 (JSON)