mlsanityData to 7 Oct 2026

CoLT5: Faster Long-Range Transformers with Conditional Computation

arXiv 2303.09752 ↗First posted 17 Mar 2023cs.CL cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Many natural language processing tasks benefit from long inputs, but processing long documents with Transformers is expensive -- not only due to quadratic attention complexity but also from applying feedforward and projection layers to every token. However, not all tokens are equally important, especially for longer documents. We propose CoLT5, a long-input Transformer model that builds on this intuition by employing conditional computation, devoting more resources to important tokens in both feedforward and attention layers. We show that CoLT5 achieves stronger performance than LongT5 with much faster training and inference, achieving SOTA on the long-input SCROLLS benchmark. Moreover, CoLT5 can effectively and tractably make use of extremely long inputs, showing strong gains up to 64k input length.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). Builds on 36 of its 49 references that are in the map. First posted on arXiv 17 Mar 2023.

Builds on

36 of 49 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
24 more
  1. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  6. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  7. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  8. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2022Confident Adaptive Language Modelingcited by 297
  11. 2022UL2: Unifying Language Learning Paradigmscited by 292
  12. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  13. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  14. 2022Scaling Up Models and Data with textttt5x and textttseqiocited by 187
  15. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  16. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  17. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  18. 2021KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answeringcited by 66
  19. 2022FiDO: Fusion-in-Decoder optimized for stronger performance and faster inferencecited by 31
  20. 2021Rider: Reader-Guided Passage Reranking for Open-Domain Question Answeringcited by 28
  21. 2018Adaptive Document Retrieval for Deep Question Answeringcited by 16
  22. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  23. 2022Multi-Vector Retrieval as Sparse Alignmentcited by 12
  24. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 21; 2025: 21; 2026: 10.

84 papers.

Sort:

By area:

  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  2. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  3. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2023A Comprehensive Overview of Large Language Modelscited by 639
  6. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  7. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  8. 2023Adapting Language Models to Compress Contextscited by 350
  9. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  10. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
  11. 2023Challenges and Applications of Large Language Modelscited by 251
  12. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
38 more
  1. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  2. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 186
  3. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  4. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
  5. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  6. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  7. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 129
  8. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  9. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  10. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
  11. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  12. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  13. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  14. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 32
  15. 2024DAPE: Data-Adaptive Positional Encoding for Length Extrapolationcited by 30
  16. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  17. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  18. 2024GRAM: Global Reasoning for Multi-Page VQAcited by 23
  19. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  20. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 23
  21. 2025CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehensioncited by 23
  22. 2023ByteSized32: A Corpus and Challenge Task for Generating Task-Specific World Models Expressed as Text Gamescited by 22
  23. 2023Learning to Skip for Language Modelingcited by 19
  24. 2025KV-Distill: Nearly Lossless Learnable Context Compression for LLMscited by 19
  25. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  26. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  27. 2024Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Modelscited by 13
  28. 2023Long-Horizon Dialogue Understanding for Role Identification in the Game of Avalon with Large Language Modelscited by 12
  29. 2024Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modulescited by 11
  30. 2023Optimizing Retrieval-augmented Reader Models via Token Eliminationcited by 10
  31. 2024Conditional computation in neural networks: principles and research trendscited by 10
  32. 2024DAPE V2: Process Attention Score as Feature Map for Length Extrapolationcited by 10
  33. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  34. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  35. 2023Chunk, Align, Select: A Simple Long-sequence Processing Method for Transformerscited by 9
  36. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 9
  37. 2023Drilling Down into the Discourse Structure with LLMs for Long Document Question Answeringcited by 8
  38. 2023GLIMMER: generalized late-interaction memory rerankercited by 7

50 most cited of 84 All 84 (JSON)