mlsanityData to 14 Oct 2026

MoDification: Mixture of Depths Made Easy

arXiv 2410.14268 ↗First posted 18 Oct 2024cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Long-context efficiency has recently become a trending topic in serving large language models (LLMs). And mixture of depths (MoD) is proposed as a perfect fit to bring down both latency and memory. In this paper, however, we discover that MoD can barely transform existing LLMs without costly training over an extensive number of tokens. To enable the transformations from any LLMs to MoD ones, we showcase top-k operator in MoD should be promoted to threshold-p operator, and refinement to architecture and data should also be crafted along. All these designs form our method termed MoDification. Through a comprehensive set of experiments covering model scales from 3B to 70B, we exhibit MoDification strikes an excellent balance between efficiency and effectiveness. MoDification can achieve up to ~1.2x speedup in latency and ~1.8x reduction in memory compared to original LLMs especially in long-context applications.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
28 more
  1. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  6. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  8. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  11. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  14. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  15. 2023Llemma: An Open Language Model For Mathematicscited by 388
  16. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  17. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  18. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  19. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  20. 2015Conditional Computation in Neural Networks for faster modelscited by 167
  21. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  22. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  23. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  24. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 44
  25. 2023Lifting the Curse of Capacity Gap in Distilling Language Modelscited by 25
  26. 2022Sparse Teachers Can Be Dense with Knowledgecited by 11
  27. 2022MiniDisc: Minimal Distillation Schedule for Language Model Compressioncited by 5
  28. 2023Task-agnostic Distillation of Encoder-Decoder Language Modelscited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertaintycited by 7