mlsanityData to 7 Oct 2026

CDLM: Cross-Document Language Modeling

arXiv 2101.00406 ↗First posted 2 Jan 2021cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

We introduce a new pretraining approach geared for multi-document language modeling, incorporating two key ideas into the masked language modeling self-supervised objective. First, instead of considering documents in isolation, we pretrain over sets of multiple related documents, encouraging the model to learn cross-document relationships. Second, we improve over recent long-range transformers by introducing dynamic global attention that has access to the entire input to predict masked tokens. We release CDLM (Cross-Document Language Model), a new general language model for multi-document setting that can be easily applied to downstream tasks. Our extensive analysis shows that both ideas are essential for the success of CDLM, and work in synergy to set new state-of-the-art results for several multi-text tasks. Code and models are available at https://github.com/aviclu/CDLM.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 2 Jan 2021.

Builds on

35 of 61 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
23 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  7. 2019Attention is not not Explanationcited by 629
  8. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  9. 2017End-to-end Neural Coreference Resolutioncited by 240
  10. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  11. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  12. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  13. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  14. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  15. 2020Pre-training via Paraphrasingcited by 92
  16. 2018Content-Based Citation Recommendationcited by 43
  17. 2019Revisiting Joint Modeling of Cross-document Entity and Event Coreference Resolutioncited by 43
  18. 2020Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matchingcited by 35
  19. 2018Resolving Event Coreference with Supervised Representation Learning and Clustering-Oriented Regularizationcited by 22
  20. 2020Streamlining Cross-Document Coreference Resolution: Evaluation and Modelingcited by 16
  21. 2020Paraphrasing vs Coreferring: Two Sides of the Same Coincited by 15
  22. 2019Hierarchical Document Encoder for Parallel Corpus Miningcited by 10
  23. 2020Multilevel Text Alignment with Cross-Document Attentioncited by 7

Built on by

Cited by year

Citing papers per year: 2021: 9; 2022: 2; 2023: 3; 2024: 0; 2025: 3.

17 papers.

Sort:

By area:

  1. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  2. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 209
  3. 2022Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Reviewcited by 37
  4. 2021Cross-document Coreference Resolution over Predicted Mentionscited by 29
  5. 2021Rethinking Search: Making Domain Experts out of Dilettantescited by 18
  6. 2021Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreferencecited by 16
  7. 2021SciCo: Hierarchical Cross-Document Coreference for Scientific Conceptscited by 14
  8. 2021Proposition-Level Clustering for Multi-Document Summarizationcited by 14
  9. 2021Sequential Cross-Document Coreference Resolutioncited by 12
  10. 2021Structure Inducing Pre-Trainingcited by 10
  11. 2021XCoref: Cross-document Coreference Resolution in the Wildcited by 6
  12. 2023Representation Learning via Variational Bayesian Networkscited by 3
5 more
  1. 2025MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidancecited by 3
  2. 2021Denoising Word Embeddings by Averaging in a Shared Spacecited by 2
  3. 2023PELMS: Pre-training for Effective Low-Shot Multi-Document Summarizationcited by 1
  4. 2025Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Seriescited by 1
  5. 2025An Senegalese Legal Texts Structuration Using LLM-augmented Knowledge Graphcited by 0