mlsanityData to 14 Oct 2026

TRAMS: Training-free Memory Selection for Long-range Language Modeling

arXiv 2310.15494 ↗First posted 24 Oct 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture is crucial for numerous AI models, but it still faces challenges in long-range language modeling. Though several specific transformer architectures have been designed to tackle issues of long-range dependencies, existing methods like Transformer-XL are plagued by a high percentage of ineffective memories. In this study, we present a plug-and-play strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens participating in attention calculation based on one simple metric. This strategy allows us to keep tokens that are likely to have a high attention score with the current queries and ignore the other ones. We have tested our approach on the word-level benchmark (WikiText-103) and the character-level benchmark (enwik8), and the results indicate an improvement without having additional training or adding additional parameters.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 25 of its 40 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

25 of 40 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  12. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
13 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  2. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  3. 2019Adaptive Attention Span in Transformerscited by 149
  4. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 128
  5. 2020Fast Transformers with Clustered Attentioncited by 123
  6. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  7. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  8. 2019When and Why is Document-level Context Useful in Neural Machine Translation?cited by 45
  9. 2020SMYRF: Efficient Attention using Asymmetric Clusteringcited by 43
  10. 2021ABC: Attention with Bounded-memory Controlcited by 36
  11. 2021Not All Memories are Created Equal: Learning to Forget by Expiringcited by 32
  12. 2021Hybrid Random Featurescited by 25
  13. 2020Sparsifying Transformer Models with Trainable Representation Poolingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 1.

5 papers.

  1. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  2. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  3. 2024Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Modelcited by 16
  4. 2024MemLong: Memory-Augmented Retrieval for Long Text Modelingcited by 11
  5. 2026DTOC: Dynamic Tool Output Compression for Adaptive Context Management in AI Agentscited by 0