AERO: Entropy-Guided Framework for Private LLM Inference
Not on the map. Why?
Abstract (arXiv)
Privacy-preserving computation enables language model inference directly on encrypted data yet suffers from prohibitive latency and communication overheads, primarily due to nonlinear functions. Removing nonlinearities, however, can trigger one of two failure modes restricting the potential for nonlinearity removal: entropy collapse in deeper layers, which destabilizes training, and entropic overload in early layers, causing under-utilization of attention heads. To address these challenges, we introduce AERO, an entropy-guided framework to strategically eliminates costly nonlinear operations from transformer architectures, which employs an adaptive recalibration through a head-wise entropy regularizer with learnable per-head strengths, enabling each head to adjust its entropy level while penalizing extreme entropies and fostering functional diversity through a tolerance margin. Experiments show AERO can save 3.4× communication and 1.4× latency, without any performance penalty.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 68 of its 99 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
68 of 99 references.
References · 68
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
- 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
38 more
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
- 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
- 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
- 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
- 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
- 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
- 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
- 2017A unified view of entropy-regularized Markov decision processescited by 174
- 2020Compositional Explanations of Neuronscited by 172
- 2018Understanding the impact of entropy on policy optimizationcited by 171
- 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
- 2023Stabilizing Transformer Training by Preventing Attention Entropy Collapsecited by 154
- 2024Stealing Part of a Production Language Modelcited by 152
- 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
- 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
- 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
- 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
- 2021CrypTen: Secure Multi-Party Computation Meets Machine Learningcited by 131
- 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
- 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
- 2022Block-Recurrent Transformerscited by 107
- 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
- 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 91
- 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
- 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
- 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
- 2022MPCFormer: fast, performant and private Transformer inference with MPCcited by 67
- 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
- 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 63
- 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
- 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
- 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
- 2024Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding?cited by 45
- 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
- 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 44
- 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
- 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
- 2023Simplifying Transformer Blockscited by 37
50 most cited of 68 All 68 (JSON)
Built on by
Citing papers per year: 2025: 3.
3 papers.