mlsanityData to 14 Oct 2026

DCT: Dynamic Compressive Transformer for Modeling Unbounded Sequence

arXiv 2110.04821 ↗First posted 10 Oct 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose Dynamic Compressive Transformer (DCT), a transformer-based framework for modeling the unbounded sequence. In contrast to the previous baselines which append every sentence representation to memory, conditionally selecting and appending them is a more reasonable solution to deal with unlimited long sequences. Our model uses a policy that determines whether the sequence should be kept in memory with a compressed state or discarded during the training process. With the benefits of retaining semantically meaningful sentence information in the memory system, our experiment results on Enwik8 benchmark show that DCT outperforms the previous state-of-the-art (SOTA) model.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 10 Oct 2021.

Builds on

27 of 38 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  9. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  10. 2016OpenAI Gymcited by 1,736
  11. 2020Reformer: The Efficient Transformercited by 1,668
  12. 2020Rethinking Attention with Performerscited by 1,601
15 more
  1. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  2. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  3. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  4. 2013Generating Sequences With Recurrent Neural Networkscited by 554
  5. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  6. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  7. 2019Axial Attention in Multidimensional Transformerscited by 358
  8. 2021Random Feature Attentioncited by 354
  9. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  10. 2020Sparse Sinkhorn Attentioncited by 251
  11. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  12. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  13. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  14. 2019Understanding LSTM -- a tutorial into Long Short-Term Memory Recurrent Neural Networkscited by 103
  15. 2020Sparse and Continuous Attention Mechanismscited by 39

Built on by

Not cited yet.