mlsanityData to 14 Oct 2026

Sign-In to the Lottery: Reparameterizing Sparse Training From Scratch

arXiv 2504.12801 ↗First posted 17 Apr 2025cs.LG cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 50 of its 67 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

50 of 67 references.

References · 50

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  6. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  8. 2018On Lazy Training in Differentiable Programmingcited by 675
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  10. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
38 more
  1. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  2. 2017Implicit Regularization in Matrix Factorizationcited by 322
  3. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  4. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  6. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  7. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  8. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 182
  9. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 151
  10. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  11. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  12. 2019Implicit Regularization for Optimal Sparse Recoverycited by 80
  13. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 79
  14. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  15. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 59
  16. 2021Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?cited by 52
  17. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
  18. 2023Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flowscited by 45
  19. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  20. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  21. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  22. 2023Dynamic Sparse Training with Structured Sparsitycited by 38
  23. 2022Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradientscited by 34
  24. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  25. 2021On the Existence of Universal Lottery Ticketscited by 30
  26. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  27. 2019High-Dimensional Linear Regression via Implicit Regularizationcited by 25
  28. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  29. 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
  30. 2020PHEW: Constructing Sparse Networks that Learn Fast and Generalize Well without Training Datacited by 17
  31. 2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
  32. 2024Masks, Signs, And Learning Rate Rewindingcited by 15
  33. 2024Mask in the Mirror: Implicit Sparsificationcited by 14
  34. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  35. 2024Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flowscited by 11
  36. 2023Implicit Regularization for Group Sparsitycited by 8
  37. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8
  38. 2024Implicit Bias of Mirror Flow for Shallow Neural Networks in Univariate Regressioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026Never Saddle for Reparameterized Steepest Descent as Mirror Flowcited by 4
  2. 2026Implicit Bias of Mirror Flow in Homogeneous Neural Networks: Sparse and Dense Feature Learningcited by 3
  3. 2025The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysiscited by 2
  4. 2026HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spacescited by 1
  5. 2026Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Ratecited by 0
  6. 2026SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Trainingcited by 0