mlsanityData to 14 Oct 2026

Mini-Sequence Transformer: Optimizing Intermediate Memory for Long Sequences Training

arXiv 2407.15892 ↗First posted 22 Jul 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

We introduce Mini-Sequence Transformer (MsT), a simple and effective methodology for highly efficient and accurate LLM training with extremely long sequences. MsT partitions input sequences and iteratively processes mini-sequences to reduce intermediate memory usage. Integrated with activation recomputation, it enables significant memory savings in both forward and backward passes. In experiments with the Llama3-8B model, with MsT, we measure no degradation in throughput or convergence even with 12x longer sequences than standard implementations. MsT is fully general, implementation-agnostic, and requires minimal code changes to integrate with existing LLM training frameworks. Integrated with the huggingface library, MsT successfully extends the maximum context length of Qwen, Mistral, and Gemma-2 by 12-24x.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 44 of its 66 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

44 of 66 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2023Mistral 7Bcited by 3,164
32 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  11. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  12. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  13. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  14. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  15. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  16. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  17. 2023ClimaX: A foundation model for weather and climatecited by 337
  18. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  19. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  20. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  21. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  22. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 244
  23. 2018Scaling Neural Machine Translationcited by 204
  24. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  25. 2021Self-attention Does Not Need O(n2) Memorycited by 154
  26. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  27. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  28. 2023Large Language Models are Few-Shot Health Learnerscited by 118
  29. 2021EDGAR-CORPUS: Billions of Tokens Make The World Go Roundcited by 32
  30. 2023AdaLomo: Low-memory Optimization with Adaptive Learning Ratecited by 31
  31. 2024TransformerFAM: Feedback attention is working memorycited by 22
  32. 2017Accumulated Gradient Normalizationcited by 10

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 5.

6 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  3. 2024Training Ultra Long Context Language Model with Fully Pipelined Distributed Transformercited by 6
  4. 2025StreamBP: Memory-Efficient Exact Backpropagation for Long Sequence Training of LLMscited by 2
  5. 2025Adjoint sharding for very long context training of state space modelscited by 0
  6. 2025MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Modelscited by 0