mlsanityData to 7 Oct 2026

ETC: Encoding Long and Structured Inputs in Transformers

arXiv 2004.08483 ↗First posted 17 Apr 2020cs.LG stat.MLcited by 34

Not on the map. Why?

Abstract (arXiv)

Transformer models have advanced the state of the art in many Natural Language Processing (NLP) tasks. In this paper, we present a new Transformer architecture, Extended Transformer Construction (ETC), that addresses two key challenges of standard Transformer architectures, namely scaling input length and encoding structured inputs. To scale attention to longer inputs, we introduce a novel global-local attention mechanism between global tokens and regular input tokens. We also show that combining global-local attention with relative position encodings and a Contrastive Predictive Coding (CPC) pre-training objective allows ETC to encode structured inputs. We achieve state-of-the-art results on four natural language datasets requiring long and/or structured inputs.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 13; 2023: 7; 2024: 2; 2025: 2; 2026: 3.

34 papers.

Sort:

By area:

  1. 2021Focal Self-attention for Local-Global Interactions in Vision Transformerscited by 286
  2. 2022Long Range Language Modeling via Gated State Spacescited by 265
  3. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  4. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  5. 2022Recurrent Memory Transformercited by 197
  6. 2021Thinking Like Transformerscited by 182
  7. 2021A Dataset for Answering Time-Sensitive Questionscited by 150
  8. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  9. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  10. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  11. 2022Learning to Reconstruct Missing Data from Spatiotemporal Graphs with Sparse Observationscited by 64
  12. 2022CMT-DeepLab: Clustering Mask Transformers for Panoptic Segmentationcited by 60
22 more
  1. 2022Investigating Efficiently Extending Transformers for Long Input Summarizationcited by 43
  2. 2022Fine-tuning Image Transformers using Learnable Memorycited by 34
  3. 2024CaDA: Cross-Problem Routing Solver with Constraint-Aware Dual-Attentioncited by 23
  4. 2023FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extractioncited by 20
  5. 2021On Learning the Transformer Kernelcited by 14
  6. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  7. 2022kMaX-DeepLab: k-means Mask Transformercited by 12
  8. 2022ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attentioncited by 6
  9. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 5
  10. 2022Global memory transformer for processing long documentscited by 4
  11. 2023MUX-PLMs: Data Multiplexing for High-throughput Language Modelscited by 4
  12. 2023A Unified Framework for Generative Data Augmentation: A Comprehensive Surveycited by 4
  13. 2025Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trendscited by 4
  14. 2022Reasoning over Logically Interacted Conditions for Question Answeringcited by 3
  15. 2022Efficient Representation Learning via Adaptive Context Poolingcited by 2
  16. 2023Vision Big Bird: Random Sparsification for Full Attentioncited by 2
  17. 2026Prefix Sliding for efficient test-time scalingcited by 2
  18. 2023Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Modelcited by 1
  19. 2023GVdoc: Graph-based Visual Document Classificationcited by 1
  20. 2026An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammarscited by 1
  21. 2021VAULT: VAriable Unified Long Text Representation for Machine Reading Comprehensioncited by 0
  22. 2026Lossy Compressive Text Autoencoderscited by 0