mlsanityData to 7 Oct 2026

Tiny Transformers Excel at Sentence Compression

arXiv 2410.23510 ↗First posted 30 Oct 2024cs.LG cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

16 of 21 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 161
  11. 2020Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Taskscited by 136
  12. 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
4 more
  1. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  2. 2020The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizescited by 48
  3. 2021Addressing "Documentation Debt" in Machine Learning Research: A Retrospective Datasheet for BookCorpuscited by 46
  4. 2018DeepZip: Lossless Data Compression using Recurrent Neural Networkscited by 32

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Small Language Models are the Future of Agentic AIcited by 254
  2. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 51