mlsanityData to 7 Oct 2026

First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models

arXiv 2408.11393 ↗First posted 21 Aug 2024cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Dynamic activation (DA) techniques, such as DejaVu and MoEfication, have demonstrated their potential to significantly enhance the inference efficiency of large language models (LLMs). However, these techniques often rely on ReLU activation functions or require additional parameters and training to maintain performance. This paper introduces a training-free Threshold-based Dynamic Activation(TDA) method that leverage sequence information to exploit the inherent sparsity of models across various architectures. This method is designed to accelerate generation speed by 18-25% without significantly compromising task performance, thereby addressing the limitations of existing DA techniques. Moreover, we delve into the root causes of LLM sparsity and theoretically analyze two of its critical features: history-related activation uncertainty and semantic-irrelevant activation inertia. Our comprehensive analyses not only provide a robust theoretical foundation for DA methods but also offer valuable insights to guide future research in optimizing LLMs for greater efficiency and effectiveness.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 35 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

25 of 35 references.

References · 25

  1. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  10. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  11. 2018CoQA: A Conversational Question Answering Challengecited by 674
  12. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 378
13 more
  1. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  2. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  3. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  5. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  6. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  7. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  8. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 41
  9. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  10. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 20
  11. 2023Model-Based Control with Sparse Neural Dynamicscited by 16
  12. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 15
  13. 2024Dynamic Activation Pitfalls in LLaMA Models: An Empirical Studycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  2. 2025Hidden Dynamics of Massive Activations in Transformer Trainingcited by 10
  3. 2025CE-LoRA: Computation-Efficient LoRA Fine-Tuning for Language Modelscited by 5
  4. 2025WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inferencecited by 2
  5. 2025GLASS: Global-Local Aggregation for Inference-time Sparsification of LLMscited by 1
  6. 2026SVD Contextual Sparsity Predictors for Fast LLM Inferencecited by 0