mlsanityData to 14 Oct 2026

MOYU: A Theoretical Study on Massive Over-activation Yielded Uplifts in LLMs

arXiv 2406.12569 ↗First posted 18 Jun 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Massive Over-activation Yielded Uplifts(MOYU) is an inherent property of large language models, and dynamic activation(DA) based on the MOYU property is a clever yet under-explored strategy designed to accelerate inference in these models. Existing methods that utilize MOYU often face a significant 'Impossible Trinity': struggling to simultaneously maintain model performance, enhance inference speed, and extend applicability across various architectures. Due to the theoretical ambiguities surrounding MOYU, this paper elucidates the root cause of the MOYU property and outlines the mechanisms behind two primary limitations encountered by current DA methods: 1) history-related activation uncertainty, and 2) semantic-irrelevant activation inertia. Our analysis not only underscores the limitations of current dynamic activation strategies within large-scale LLaMA models but also proposes opportunities for refining the design of future sparsity schemes.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 20 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

11 of 20 references.

References · 11

  1. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  4. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  5. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 201
  6. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 122
  7. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 45
  8. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Modelscited by 42
  9. 2018Accelerating Convolutional Neural Networks via Activation Map Compressioncited by 34
  10. 2024Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generationcited by 21
  11. 2024Learn To be Efficient: Build Structured Sparsity in Large Language Modelscited by 20

Built on by

Not cited yet.