mlsanityData to 7 Oct 2026

FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment

arXiv 2602.02680 ↗First posted 2 Feb 2026cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

The growing scale of deep neural networks, encompassing large language models (LLMs) and vision transformers (ViTs), has made training from scratch prohibitively expensive and deployment increasingly costly. These models are often used as computational monoliths with fixed cost, hindering adaptive deployment across different cost budgets. We argue that nested components, ordered by importance, can be extracted from pretrained models and selectively activated within the available computational budget. To this end, our proposed FlexRank method leverages low-rank weight decomposition with nested, importance-based consolidation to extract submodels of increasing capabilities. Our approach enables a "train-once, deploy-everywhere" paradigm offering a graceful trade-off between cost and performance without training from scratch for each budget - advancing practical deployment of large models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 43 of its 54 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

43 of 54 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  12. 2025Kimi K2: Open Agentic Intelligencecited by 523
31 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  2. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  3. 2018Slimmable Neural Networkscited by 301
  4. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  5. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  6. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  7. 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
  8. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  9. 2021Accelerating Sparse Deep Neural Networkscited by 213
  10. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  11. 2022Language model compression with weighted low-rank factorizationcited by 192
  12. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  13. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  14. 2024Nemotron-4 340B Technical Reportcited by 143
  15. 2024LoRA-GA: Low-Rank Adaptation with Gradient Approximationcited by 134
  16. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 123
  17. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 98
  18. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  19. 2025SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compressioncited by 63
  20. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  21. 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
  22. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  23. 2024Flextron: Many-in-One Flexible Large Language Modelcited by 42
  24. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  25. 2022SPDY: Accurate Pruning with Speedup Guaranteescited by 34
  26. 2025LoFT: Low-Rank Adaptation That Behaves Like Full Fine-Tuningcited by 13
  27. 2024Recurrent Early Exits for Federated Learning with Heterogeneous Clientscited by 11
  28. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  29. 2025A3 : an Analytical Low-Rank Approximation Framework for Attentioncited by 7
  30. 2025Deep Hierarchical Learning with Nested Subspace Networks for Large Language Modelscited by 5
  31. 2025Choose Your Model Size: Any Compression of Large Language Models Without Re-Computationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning in the Null Space: Small Singular Values for Continual Learningcited by 1
  2. 2026KV-Kaizen: Learning Context-Adaptive Cache Compression Choicescited by 0