mlsanityData to 7 Oct 2026

Extending Input Contexts of Language Models through Training on Segmented Sequences

arXiv 2310.14633 ↗First posted 23 Oct 2023cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Effectively training language models on long inputs poses many technical challenges. As a cost consideration, languages models are pretrained on a fixed sequence length before being adapted to longer sequences. We explore various methods for adapting models to longer inputs by training on segmented sequences and an interpolation-based method for extending absolute positional embeddings. We develop a training procedure to extend the input context size of pretrained models with no architectural changes and no additional memory costs than training on the original input lengths. By sub-sampling segments from long inputs while maintaining their original position the model is able to learn new positional interactions. Our method benefits both models trained with absolute positional embeddings, by extending their input contexts, as well as popular relative positional embedding methods showing a reduced perplexity on sequences longer than they were trained on. We demonstrate our method can extend input contexts by a factor of 4x while improving perplexity.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

35 of 47 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2020Reformer: The Efficient Transformercited by 1,658
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
23 more
  1. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  2. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  3. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  4. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  5. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  6. 2023The Impact of Positional Encoding on Length Generalization in Transformerscited by 319
  7. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  8. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  9. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  10. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  11. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  12. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 148
  13. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  14. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 114
  15. 2022KERPLE: Kernelized Relative Positional Embedding for Length Extrapolationcited by 84
  16. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  17. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  18. 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
  19. 2023In-Context Learning with Many Demonstration Examplescited by 39
  20. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  21. 2021Simple Local Attentions Remain Competitive for Long-Context Taskscited by 25
  22. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  23. 2023A Frustratingly Easy Improvement for Position Embeddings via Random Paddingcited by 4

Built on by

Not cited yet.