mlsanityData to 14 Oct 2026

Harnessing Attention Mechanisms: Efficient Sequence Reduction using Attention-based Autoencoders

arXiv 2310.14837 ↗First posted 23 Oct 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Many machine learning models use the manipulation of dimensions as a driving force to enable models to identify and learn important features in data. In the case of sequential data this manipulation usually happens on the token dimension level. Despite the fact that many tasks require a change in sequence length itself, the step of sequence length reduction usually happens out of necessity and in a single step. As far as we are aware, no model uses the sequence length reduction step as an additional opportunity to tune the models performance. In fact, sequence length manipulation as a whole seems to be an overlooked direction. In this study we introduce a novel attention-based method that allows for the direct manipulation of sequence lengths. To explore the method's capabilities, we employ it in an autoencoder model. The autoencoder reduces the input sequence to a smaller sequence in latent space. It then aims to reproduce the original sequence from this reduced form. In this setting, we explore the methods reduction performance for different input and latent sequence lengths. We are able to show that the autoencoder retains all the significant information when reducing the original sequence to half its original size. When reducing down to as low as a quarter of its original size, the autoencoder is still able to reproduce the original sequence with an accuracy of around 90%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  12. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 463
10 more
  1. 2021Learned Token Pruning for Transformerscited by 139
  2. 2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
  3. 2020Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Searchcited by 76
  4. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 62
  5. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  6. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  7. 2022SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samplescited by 39
  8. 2021Finetuning Pretrained Transformers into Variational Autoencoderscited by 14
  9. 2019Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embeddingcited by 12
  10. 2023Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCEcited by 7

Built on by

Not cited yet.