mlsanityData to 14 Oct 2026

TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training

arXiv 2511.03983 ↗First posted 6 Nov 2025cs.LG math.OCcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce TwIST, a distributed training framework for efficient large language model (LLM) sparsification. TwIST trains multiple subnetworks in parallel, periodically aggregates their parameters, and resamples new subnetworks during training. This process identifies high-quality subnetworks ("golden tickets") without requiring post-training procedures such as calibration or Hessian-based recovery. As a result, TwIST enables zero-cost pruning at deployment time while achieving perplexity competitive with state-of-the-art post-training sparsification methods. The benefits are most pronounced under aggressive sparsity (e.g., 50%+), where TwIST significantly outperforms baseline methods; for example, reaching 23.14 PPL compared to 31.64 for the closest prior approach. Unlike unstructured pruning, TwIST produces structured, dense matrices that offer practical inference speedups and memory reductions on commodity hardware (e.g., CPUs) that do not support efficient sparse computation. TwIST provides an efficient training-time path to deployable sparse LLMs without additional fine-tuning or recovery overhead.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 6 Nov 2025.

Builds on

47 of 71 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2022Emergent Abilities of Large Language Modelscited by 2,512
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
35 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2020On Layer Normalization in the Transformer Architecturecited by 922
  5. 2019Are Sixteen Heads Really Better than One?cited by 816
  6. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  7. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  8. 2018Rethinking the Value of Network Pruningcited by 599
  9. 2020What is the State of Neural Network Pruning?cited by 583
  10. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  12. 2019The State of Sparsity in Deep Neural Networkscited by 476
  13. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  14. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  15. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  16. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  17. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 182
  18. 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
  19. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  20. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  21. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  22. 2022GPT Takes the Bar Examcited by 65
  23. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  24. 2023Intriguing Properties of Quantization at Scalecited by 49
  25. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  26. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  27. 2023Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddingscited by 23
  28. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  29. 2024Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Modelscited by 18
  30. 2025Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generationcited by 16
  31. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  32. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  33. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8
  34. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4
  35. 2025Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retrainingcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Expertscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating LLM Inference with Flexible N:M Sparsity via A Fully Digital Compute-in-Memory Accelerator
  2. 2026PrunePath: Towards Highly Structured Sparse Language Models
  3. 2025CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models
  4. 2025Amber Pruner: Leveraging N:M Activation Sparsity for Efficient Prefill in Large Language Models
  5. 2021Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing
  6. 2024FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
  7. 2021NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
  8. 2023One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models