mlsanityData to 14 Oct 2026

ElastiFormer: Learned Redundancy Reduction in Transformer via Self-Distillation

arXiv 2411.15281 ↗First posted 22 Nov 2024cs.LG cs.AIcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

We introduce ElastiFormer, a post-training technique that adapts pretrained Transformer models into an elastic counterpart with variable inference time compute. ElastiFormer introduces small routing modules (as low as .00006% additional trainable parameters) to dynamically selects subsets of network parameters and input tokens to be processed by each layer of the pretrained network in an inputdependent manner. The routing modules are trained using self-distillation losses to minimize the differences between the output of the pretrained-model and their elastic counterparts. As ElastiFormer makes no assumption regarding the modality of the pretrained Transformer model, it can be readily applied to all modalities covering causal language modeling, image modeling as well as visual-language modeling tasks. We show that 20% to 50% compute saving could be achieved for different components of the transformer layer, which could be further reduced by adding very low rank LoRA weights (rank 1) trained via the same distillation objective. Finally, by comparing routing trained on different subsets of ImageNet, we show that ElastiFormer is robust against the training domain.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 35 of its 86 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

35 of 86 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul

Other references · 34

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2024Mixtral of Expertscited by 1,823
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  9. 2022Token Merging: Your ViT But Fastercited by 1,021
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
22 more
  1. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
  2. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  3. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 415
  4. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  5. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  6. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  7. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 151
  8. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  9. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  10. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  11. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  12. 2024What Matters in Transformers? Not All Attention is Neededcited by 84
  13. 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 82
  14. 2022Unified Visual Transformer Compressioncited by 79
  15. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  16. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 76
  17. 2022MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptationcited by 37
  18. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  19. 2021Sparse Progressive Distillation: Resolving Overfitting under Pretrain-and-Finetune Paradigmcited by 21
  20. 2021Anytime Dense Prediction with Confidence Adaptivitycited by 18
  21. 2022Self-Distillation for Further Pre-training of Transformerscited by 9
  22. 2020Analyzing Redundancy in Pretrained Transformer Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026UNIFERENCE: A Discrete Event Simulation Framework for Developing Distributed AI Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  2. 2024The Need for Speed: Pruning Transformers with One Recipe
  3. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
  4. 2025EcoSpa: Efficient Transformer Training with Coupled Sparsity
  5. 2021Mesa: A Memory-saving Training Framework for Transformers
  6. 2022HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression
  7. 2024From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers
  8. 2023A Survey of Techniques for Optimizing Transformer Inference