mlsanityData to 14 Oct 2026

PruMUX: Augmenting Data Multiplexing with Model Compression

arXiv 2305.14706 ↗First posted 24 May 2023cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

As language models increase in size by the day, methods for efficient inference are critical to leveraging their capabilities for various applications. Prior work has investigated techniques like model pruning, knowledge distillation, and data multiplexing to increase model throughput without sacrificing accuracy. In this paper, we combine two such methods -- structured pruning and data multiplexing -- to compound the speedup gains obtained by either method. Our approach, PruMUX, obtains up to 7.5-29.5X throughput improvement over BERT-base model with accuracy threshold from 80% to 74%. We further study various combinations of parameters (such as sparsity and multiplexing factor) in the two techniques to provide a comprehensive analysis of the tradeoff between accuracy and throughput in the resulting models. We then propose Auto-PruMUX, a meta-level model that can predict the high-performance parameters for pruning and multiplexing given a desired accuracy loss budget, providing a practical method to leverage the combination effectively.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 41 of its 69 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

41 of 69 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  12. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
29 more
  1. 2019Are Sixteen Heads Really Better than One?cited by 816
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  3. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  4. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  5. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  6. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  7. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  8. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  9. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 251
  10. 2019Structured Pruning of Large Language Modelscited by 203
  11. 2020Training independent subnetworks for robust predictioncited by 177
  12. 2021Block Pruning For Faster Transformerscited by 177
  13. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  14. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  15. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  16. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  17. 2018Predicting the Computational Cost of Deep Learning Modelscited by 49
  18. 2019Overparameterized Neural Networks Implement Associative Memorycited by 45
  19. 2017Block-Sparse Recurrent Neural Networkscited by 39
  20. 2017Exploring the Regularity of Sparse Structure in Convolutional Neural Networkscited by 37
  21. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  22. 2017NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networkscited by 30
  23. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 21
  24. 2017Learning Intrinsic Sparse Structures within Long Short-Term Memorycited by 18
  25. 2022DataMUX: Data Multiplexing for Neural Networkscited by 18
  26. 2017HyperPower: Power- and Memory-Constrained Hyper-Parameter Optimization for Neural Networkscited by 11
  27. 2022Accelerating DNN Training with Structured Data Gradient Pruningcited by 8
  28. 2020Performance Prediction for Convolutional Neural Networks in Edge Devicescited by 5
  29. 2023MUX-PLMs: Data Multiplexing for High-throughput Language Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1.

3 papers.

  1. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 90
  2. 2023MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superpositioncited by 14
  3. 2023Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modulescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Pruning Laws for Large Language Models
  2. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models
  3. 2023How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark
  4. 2022Combining Compressions for Multiplicative Size Scaling on Natural Language Tasks
  5. 2026Frequency Matters: Fast Model-Agnostic Data Curation for Pruning and Quantization
  6. 2024Aggressive Post-Training Compression on Extremely Large Language Models
  7. 2023Just CHOP: Embarrassingly Simple LLM Compression
  8. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models