# Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation arXiv: https://arxiv.org/abs/2402.02855 ยท HTML: https://mlsanity.com/paper/2402.02855/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 38 of its 62 references that are in the map. - First posted on arXiv 5 Feb 2024. ## Builds on (most cited first, 30 of 38) - [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016) - [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015) - [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018) - [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018) - [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020) - [LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation](https://mlsanity.com/paper/2002.02126/) (2020) - [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017) - [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016) - [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017) - [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019) - [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020) - [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023) - [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018) - [Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks](https://mlsanity.com/paper/1909.11957/) (2019) - [A Unified Lottery Ticket Hypothesis for Graph Neural Networks](https://mlsanity.com/paper/2102.06790/) (2021) - [Winning the Lottery with Continuous Sparsification](https://mlsanity.com/paper/1912.04427/) (2019) - [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019) - [UltraGCN: Ultra Simplification of Graph Convolutional Networks for Recommendation](https://mlsanity.com/paper/2110.15114/) (2021) - [Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win](https://mlsanity.com/paper/2010.03533/) (2020) - [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018) - [AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations](https://mlsanity.com/paper/2002.11252/) (2020) - [Learnable Embedding Sizes for Recommender Systems](https://mlsanity.com/paper/2101.07577/) (2021) - [Outer Product-based Neural Collaborative Filtering](https://mlsanity.com/paper/1808.03912/) (2018) - [Sparsity Winning Twice: Better Robust Generalization from More Efficient Training](https://mlsanity.com/paper/2202.09844/) (2022) - [Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift](https://mlsanity.com/paper/2211.02843/) (2022) - [GANs Can Play Lottery Tickets Too](https://mlsanity.com/paper/2106.00134/) (2021) - [AutoML for Deep Recommender Systems: A Survey](https://mlsanity.com/paper/2203.13922/) (2022) - [Topology Distillation for Recommender System](https://mlsanity.com/paper/2106.08700/) (2021) - [Cross-Task Knowledge Distillation in Multi-Task Recommendation](https://mlsanity.com/paper/2202.09852/) (2022) - [Bidirectional Distillation for Top-K Recommender System](https://mlsanity.com/paper/2106.02870/) (2021) ## Built on by (most cited first, 7 of 7) - [Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs](https://mlsanity.com/paper/2410.23875/) (2024) - [Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation](https://mlsanity.com/paper/2501.14269/) (2025) - [Active Large Language Model-based Knowledge Distillation for Session-based Recommendation](https://mlsanity.com/paper/2502.15685/) (2024) - [A Unified Invariant Learning Framework for Graph Classification](https://mlsanity.com/paper/2501.12595/) (2025) - [A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems](https://mlsanity.com/paper/2406.17335/) (2024) - [On-device Content-based Recommendation with Single-shot Embedding Pruning: A Cooperative Game Perspective](https://mlsanity.com/paper/2411.13052/) (2024) - [Guided Navigation in Knowledge-Dense Environments: Structured Semantic Exploration with Guidance Graphs](https://mlsanity.com/paper/2508.10012/) (2025) ## Abstract (arXiv) In the realm of deep learning-based recommendation systems, the increasing computational demands, driven by the growing number of users and items, pose a significant challenge to practical deployment. This challenge is primarily twofold: reducing the model size while effectively learning user and item representations for efficient recommendations. Despite considerable advancements in model compression and architecture search, prevalent approaches face notable constraints. These include substantial additional computational costs from pre-training/re-training in model compression and an extensive search space in architecture design. Additionally, managing complexity and adhering to memory constraints is problematic, especially in scenarios with strict time or space limitations. Addressing these issues, this paper introduces a novel learning paradigm, Dynamic Sparse Learning (DSL), tailored for recommendation models. DSL innovatively trains a lightweight sparse model from scratch, periodically evaluating and dynamically adjusting each weight's significance and the model's sparsity distribution during the training. This approach ensures a consistent and minimal parameter budget throughout the full learning lifecycle, paving the way for "end-to-end" efficiency from training to inference. Our extensive experimental results underline DSL's effectiveness, significantly reducing training and inference costs while delivering comparable recommendation performance.