mlsanityData to 14 Oct 2026

DeCAL Tokenwise Compression

arXiv 2508.08514 ↗First posted 11 Aug 2025cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper introduces DeCAL, a new method for tokenwise compression. DeCAL uses an encoder-decoder language model pretrained with denoising to learn to produce high-quality, general-purpose compressed representations from the encoder. DeCAL applies small modifications to the encoder, with the emphasis on maximizing compression quality, even at the expense of compute. We show that DeCAL at 2x compression can match uncompressed on several downstream tasks, with usually only a minor dropoff in metrics up to 8x compression, among question-answering, summarization, and multi-vector retrieval tasks. DeCAL offers significant savings where pre-computed dense representations can be utilized, and we believe the approach can be further developed to be more broadly applicable.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

29 of 32 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  12. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
17 more
  1. 2020Efficient Transformers: A Surveycited by 790
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  3. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  4. 2023Adapting Language Models to Compress Contextscited by 356
  5. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  6. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  7. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  8. 2020Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processingcited by 162
  9. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  10. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  11. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 102
  12. 2022PLAID: An Efficient Engine for Late Interaction Retrievalcited by 101
  13. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  14. 2024A Survey on Transformer Compressioncited by 46
  15. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  16. 2023Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your computecited by 17
  17. 2022Multi-Vector Retrieval as Sparse Alignmentcited by 12

Built on by

Not cited yet.