mlsanityData to 7 Oct 2026

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length

arXiv 2404.08801 ↗First posted 12 Apr 2024cs.LG cs.CLcited by 46

Not on the map. Why?

Abstract (arXiv)

The quadratic complexity and weak length extrapolation of Transformers limits their ability to scale to long sequences, and while sub-quadratic solutions like linear attention and state space models exist, they empirically underperform Transformers in pretraining efficiency and downstream task accuracy. We introduce Megalodon, a neural architecture for efficient sequence modeling with unlimited context length. Megalodon inherits the architecture of Mega (exponential moving average with gated attention), and further introduces multiple technical components to improve its capability and stability, including complex exponential moving average (CEMA), timestep normalization layer, normalized attention mechanism and pre-norm with two-hop residual configuration. In a controlled head-to-head comparison with Llama2, Megalodon achieves better efficiency than Transformer in the scale of 7 billion parameters and 2 trillion training tokens. Megalodon reaches a training loss of 1.70, landing mid-way between Llama2-7B (1.75) and 13B (1.67). Code: https://github.com/XuezheMax/megalodon

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Builds on 56 of its 77 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

56 of 77 references.

References · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2023Mistral 7Bcited by 3,065
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2018Group Normalizationcited by 1,514
  10. 2019Root Mean Square Layer Normalizationcited by 1,425
  11. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  12. 2020On Layer Normalization in the Transformer Architecturecited by 906
  13. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  14. 2020Efficient Transformers: A Surveycited by 784
  15. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  16. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  17. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  18. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  19. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  20. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  21. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  22. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  23. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  24. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  25. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  26. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  27. 2020Query-Key Normalization for Transformerscited by 266
  28. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  29. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  30. 2022Transformer Quality in Linear Timecited by 189
  31. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  32. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  33. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  34. 2017Searching for Activation Functionscited by 158
  35. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  36. 2022Block-Recurrent Transformerscited by 107
  37. 2022Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?cited by 103
  38. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 13; 2026: 9.

46 papers.

Sort:

By area:

  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  3. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  4. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 167
  5. 2025Test-Time Training Done Rightcited by 121
  6. 2024Hyper-Connectionscited by 81
  7. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  8. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  9. 2025The Curse of Depth in Large Language Modelscited by 58
  10. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  11. 2025Rope to Nope and Back Again: A New Hybrid Attention Strategycited by 46
  12. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
34 more
  1. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  2. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  3. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  4. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 29
  5. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  6. 2026MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokenscited by 20
  7. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  8. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  9. 2024ViG: Linear-complexity Visual Sequence Learning with Gated Linear Attentioncited by 13
  10. 2024CiteAssist: A System for Automated Preprint Citation and BibTeX Generationcited by 11
  11. 2024Efficient Streaming LLM for Speech Recognitioncited by 10
  12. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  13. 2024Toward Conversational Agents with Context and Time Sensitive Long-term Memorycited by 8
  14. 2024EEG Emotion Copilot: Optimizing Lightweight LLMs for Emotional EEG Interpretation with Assisted Medical Record Generationcited by 6
  15. 2024On the Power of Convolution Augmented Transformercited by 5
  16. 2025From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generationcited by 4
  17. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  18. 2025Local Linear Attention: An Optimal Interpolation of Linear and Softmax Attention For Test-Time Regressioncited by 4
  19. 2024Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyondcited by 3
  20. 2025Interactive Data Harmonization with LLM Agents: Opportunities and Challengescited by 3
  21. 2025Dynamic Chunking and Selection for Reading Comprehension of Ultra-Long Context in Large Language Modelscited by 3
  22. 2025RAT: Bridging RNN Efficiency and Attention Accuracy via Chunk-based Sequence Modelingcited by 3
  23. 2025MemMamba: Rethinking Memory Patterns in State Space Modelcited by 3
  24. 2025Elucidating the Design Space of Decay in Linear Attentioncited by 1
  25. 2026The Impossibility Triangle of Long-Context Modelingcited by 1
  26. 2026Blurry Window Attentioncited by 1
  27. 2026State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Trackingcited by 1
  28. 2024SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLMcited by 0
  29. 2024Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Rolescited by 0
  30. 2024How much do contextualized representations encode long-range context?cited by 0
  31. 2026Gecko: An Efficient Neural Architecture Inherently Processing Sequences with Arbitrary Lengthscited by 0
  32. 2026Stacked from One: Multi-Scale Self-Injection for Context Window Extensioncited by 0
  33. 2026Cubit: Token Mixer with Kernel Ridge Regressioncited by 0
  34. 2026Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extensioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
  2. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
  3. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mamba
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline
  5. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression
  6. 2025Overflow Prevention Enhances Long-Context Recurrent LLMs
  7. 2025LMDeploy Accelerates Mixed-Precision LLM Inference with TurboMind
  8. 2025AccLLM: Accelerating Long-Context LLM Inference Via Algorithm-Hardware Co-Design