mlsanityData to 7 Oct 2026

Flextron: Many-in-One Flexible Large Language Model

arXiv 2406.10260 ↗First posted 11 Jun 2024cs.CL cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical. In this paper, we introduce Flextron, a network architecture and post-training model optimization framework supporting flexible model deployment. The Flextron architecture utilizes a nested elastic structure to rapidly adapt to specific user-defined latency and accuracy targets during inference with no additional fine-tuning required. It is also input-adaptive, and can automatically route tokens through its sub-networks for improved performance and efficiency. We present a sample-efficient training method and associated routing algorithms for systematically transforming an existing trained LLM into a Flextron model. We evaluate Flextron on the GPT-3 and LLama-2 family of LLMs, and demonstrate superior performance over multiple end-to-end trained variants and other state-of-the-art elastic networks, all with a single pretraining run that consumes a mere 7.63% tokens compared to original pretraining.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

42 of 53 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
30 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  2. 2024Mixtral of Expertscited by 1,785
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  4. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  8. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  9. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  10. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  11. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  12. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  13. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
  14. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  15. 2018Slimmable Neural Networkscited by 301
  16. 2022Matryoshka Representation Learningcited by 268
  17. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  18. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  19. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  20. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  21. 2024LaCo: Large Language Model Pruning via Layer Collapsecited by 124
  22. 2019Single-Path NAS: Designing Hardware-Efficient ConvNets in less than 4 Hourscited by 87
  23. 2021Dynamic Slimmable Networkcited by 83
  24. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  25. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  26. 2021Global Vision Transformer Pruning with Hessian-Aware Saliencycited by 69
  27. 2023Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformerscited by 59
  28. 2024Nemotron-4 15B Technical Reportcited by 36
  29. 2023Robust Mixture-of-Expert Training for Convolutional Neural Networkscited by 30
  30. 2023Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?cited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 16.

42 papers.

Sort:

By area:

  1. 2025Small Language Models are the Future of Agentic AIcited by 254
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  4. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  5. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  6. 2025DarwinLM: Evolutionary Structured Pruning of Large Language Modelscited by 16
  7. 2025Cost-Aware Contrastive Routing for LLMscited by 16
  8. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  9. 2025Learning to Inference Adaptively for Multimodal Large Language Modelscited by 12
  10. 2026ERNIE 5.0 Technical Reportcited by 9
  11. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  12. 2025MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeXcited by 7
30 more
  1. 2024MatMamba: A Matryoshka State Space Modelcited by 6
  2. 2024Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Modelscited by 6
  3. 2025DLP: Dynamic Layerwise Pruning in Large Language Modelscited by 6
  4. 2025ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inferencecited by 6
  5. 2025Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilizationcited by 6
  6. 2025UniQL: Unified Quantization and Low-rank Compression for Adaptive Edge LLMscited by 6
  7. 2025EA-ViT: Efficient Adaptation for Elastic Vision Transformercited by 5
  8. 2025Deep Hierarchical Learning with Nested Subspace Networks for Large Language Modelscited by 5
  9. 2025Masked Generative Nested Transformers with Decode Time Scalingcited by 4
  10. 2025Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMscited by 4
  11. 2025FlexGS: Train Once, Deploy Everywhere with Many-in-One Flexible 3D Gaussian Splattingcited by 3
  12. 2024Mixture of Hidden-Dimensions Transformercited by 2
  13. 2025DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformerscited by 2
  14. 2026FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deploymentcited by 2
  15. 2026Star Elastic: Many-in-One Reasoning LLMs with Efficient Budget Controlcited by 2
  16. 2025AdaPerceiver: Transformers with Adaptive Width, Depth, and Tokenscited by 1
  17. 2026SALAAD: Sparse And Low-Rank Adaptation via ADMM for Large Language Model Inferencecited by 1
  18. 2025MoE-PHDS: One MoE checkpoint for flexible runtime sparsitycited by 0
  19. 2025E3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Modelscited by 0
  20. 2026EPAS: Efficient Training with Progressive Activation Sharingcited by 0
  21. 2026TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlationcited by 0
  22. 2026MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptationcited by 0
  23. 2026Do Vision Transformers Need All-to-All Attention? Global Communication Through Elastic Learned Corescited by 0
  24. 2026ML-Embed: Inclusive and Efficient Embeddings for a Multilingual Worldcited by 0
  25. 2026Slimmable ConvNeXt: Width-Adaptive Inference for Efficient Multi-Device Deploymentcited by 0
  26. 2026LLM Compression with Jointly Optimizing Architectural and Quantization choicescited by 0
  27. 2026FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Modelscited by 0
  28. 2026Matryoshka Language Model Suitescited by 0
  29. 2026Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inferencecited by 0
  30. 2026Telescopic Language Modelscited by 0