mlsanityData to 7 Oct 2026

SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM

arXiv 2406.06571 ↗First posted 3 Jun 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

While Large Language Models (LLMs) have achieved remarkable success in various fields, the efficiency of training and inference remains a major challenge. To address this issue, we propose SUBLLM, short for Subsampling-Upsampling-Bypass Large Language Model, an innovative architecture that extends the core decoder-only framework by incorporating subsampling, upsampling, and bypass modules. The subsampling modules are responsible for shortening the sequence, while the upsampling modules restore the sequence length, and the bypass modules enhance convergence. In comparison to LLaMA, the proposed SUBLLM exhibits significant enhancements in both training and inference speeds as well as memory usage, while maintaining competitive few-shot performance. During training, SUBLLM increases speeds by 26% and cuts memory by 10GB per GPU. In inference, it boosts speeds by up to 37% and reduces memory by 1GB per GPU. The training and inference speeds can be enhanced by 34% and 52% respectively when the context window is expanded to 8192. Our code is available at https://github.com/XiaoMi/subllm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 55 of its 64 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

55 of 64 references.

References · 55

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  13. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  14. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  15. 2020Efficient Transformers: A Surveycited by 784
  16. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  17. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  18. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  19. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  20. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  21. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  22. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  23. 2022Large Language Models Are Reasoning Teacherscited by 318
  24. 2022Teaching Small Language Models to Reasoncited by 279
  25. 2023A Survey on Model Compression for Large Language Modelscited by 263
  26. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  27. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  28. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  29. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  30. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  31. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 135
  32. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  33. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Modelscited by 112
  34. 2021An Attention Free Transformercited by 111
  35. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  36. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  37. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  38. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.