mlsanityData to 13 Oct 2026

ERNIE-Doc: A Retrospective Long-Document Modeling Transformer

arXiv 2012.15688 ↗First posted 31 Dec 2020cs.CLcited by 31

Not on the map. Why?

Abstract (arXiv)

Transformers are not suited for processing long documents, due to their quadratically increasing memory and time consumption. Simply truncating a long document or applying the sparse attention mechanism will incur the context fragmentation problem or lead to an inferior modeling capability against comparable model sizes. In this paper, we propose ERNIE-Doc, a document-level language pretraining model based on Recurrence Transformers. Two well-designed techniques, namely the retrospective feed mechanism and the enhanced recurrence mechanism, enable ERNIE-Doc, which has a much longer effective context length, to capture the contextual information of a complete document. We pretrain ERNIE-Doc to explicitly learn the relationships among segments with an additional document-aware segment-reordering objective. Various experiments were conducted on both English and Chinese document-level tasks. ERNIE-Doc improved the state-of-the-art language modeling result of perplexity to 16.8 on WikiText-103. Moreover, it outperformed competitive pretraining models by a large margin on most language understanding tasks, such as text classification and question answering.

Key facts. Cited by 31 papers in the map (data through 13 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 5; 2022: 4; 2023: 12; 2024: 5; 2025: 2; 2026: 3.

31 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 400
  2. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 259
  3. 2022Recurrent Memory Transformercited by 199
  4. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 98
  5. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  6. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 49
  7. 2022Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generationcited by 48
  8. 2023A Survey on Long Text Modeling with Transformerscited by 47
  9. 2021ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 46
  10. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  11. 2024TransformerFAM: Feedback attention is working memorycited by 21
  12. 2022Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformercited by 20
19 more
  1. 2023Recurrent Action Transformer with Memorycited by 17
  2. 2023Large Language Models are legal but they are not: Making the case for a powerful LegalLLMcited by 12
  3. 2023Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarizationcited by 7
  4. 2024Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classificationcited by 7
  5. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusioncited by 7
  6. 2021Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Datacited by 6
  7. 2023AllSpark: A Multimodal Spatio-Temporal General Intelligence Model with Ten Modalities via Language as a Reference Frameworkcited by 6
  8. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  9. 2021PoNet: Pooling Network for Efficient Token Mixing in Long Sequencescited by 5
  10. 2026CoMeT: Collaborative Memory Transformer for Efficient Long Context Modelingcited by 5
  11. 2023MoviePuzzle: Visual Narrative Reasoning through Multimodal Order Learningcited by 1
  12. 2023S2vNTM: Semi-supervised vMF Neural Topic Modelingcited by 1
  13. 2023Labels Need Prompts Too: Mask Matching for Natural Language Understanding Taskscited by 1
  14. 2026InfoMem: Training Long-Context Memory Agents with Answer-Conditioned Information Gaincited by 1
  15. 2023A General-Purpose Multilingual Document Encodercited by 0
  16. 2023Incrementally-Computable Neural Networks: Efficient Inference for Dynamic Inputscited by 0
  17. 2024Graph-tree Fusion Model with Bidirectional Information Propagation for Long Document Classificationcited by 0
  18. 2025Personality Prediction from Life Stories using Language Modelscited by 0
  19. 2026Whole-Piece Training for Symbolic Music Language Models via Full-Horizon Compressed Recurrencecited by 0