mlsanityData to 14 Oct 2026

Text Compression for Efficient Language Generation

arXiv 2503.11426 ↗First posted 14 Mar 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the "Generative Pretrained Thoughtformer" (GPTHF), a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism. GPTHF retains GPT's architecture, modifying only token interactions via dynamic sparse attention masks. Our experiments show that GPTHF achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized GPT models in the low-size regime. This is achieved through a unique generation method that caches and reuses sentence embeddings, allowing significant portions of the input to bypass large parts of the network.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 41 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

30 of 41 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
18 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  2. 2017Decoupled Weight Decay Regularizationcited by 1,831
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  7. 2016Sequence-Level Knowledge Distillationcited by 801
  8. 2023A Comprehensive Overview of Large Language Modelscited by 648
  9. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  10. 2022Efficiently Scaling Transformer Inferencecited by 539
  11. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  12. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  13. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  14. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
  15. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  16. 2018Training Tips for the Transformer Modelcited by 85
  17. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  18. 2021Sentence Bottleneck Autoencoders from Transformer Language Modelscited by 19

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Small Language Models are the Future of Agentic AIcited by 270