mlsanityData to 14 Oct 2026

Contrastive Document Representation Learning with Graph Attention Networks

arXiv 2110.10778 ↗First posted 20 Oct 2021cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent progress in pretrained Transformer-based language models has shown great success in learning contextual representation of text. However, due to the quadratic self-attention complexity, most of the pretrained Transformers models can only handle relatively short text. It is still a challenge when it comes to modeling very long documents. In this work, we propose to use a graph attention network on top of the available pretrained Transformers model to learn document embeddings. This graph attention network allows us to leverage the high-level semantic structure of the document. In addition, based on our graph document model, we design a simple contrastive learning strategy to pretrain our models on a large amount of unlabeled corpus. Empirically, we demonstrate the effectiveness of our approaches in document classification and document retrieval tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 32 of its 46 references that are in the map. First posted on arXiv 20 Oct 2021.

Builds on

32 of 46 references.

References · 32

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2017Graph Attention Networkscited by 8,923
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
20 more
  1. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  2. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 695
  3. 2018Graph Convolutional Networks for Text Classificationcited by 345
  4. 2020Overview of the TREC 2019 deep learning trackcited by 286
  5. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  6. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  7. 2020Sparse, Dense, and Attentional Representations for Text Retrievalcited by 214
  8. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  9. 2020CERT: Contrastive Self-supervised Learning for Language Understandingcited by 169
  10. 2019Text Classification Algorithms: A Surveycited by 163
  11. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  12. 2019Hierarchical Transformers for Long Document Classificationcited by 104
  13. 2016Text Matching as Image Recognitioncited by 59
  14. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  15. 2020CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understandingcited by 41
  16. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  17. 2019Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classificationcited by 32
  18. 2020Embedding-based Zero-shot Retrieval through Query Generationcited by 24
  19. 2019WIKIR: A Python toolkit for building a large-scale Wikipedia-based English Information Retrieval Datasetcited by 8
  20. 2021Unsupervised Document Embedding via Contrastive Augmentationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 0; 2025: 1; 2026: 1.

6 papers.

  1. 2022Recurrent Memory Transformercited by 203
  2. 2022SciRepEval: A Multi-Format Benchmark for Scientific Document Representationscited by 129
  3. 2023Finding the Law: Enhancing Statutory Article Retrieval via Graph Neural Networkscited by 9
  4. 2023Coreference Graph Guidance for Mind-Map Generationcited by 2
  5. 2025Rethinking Graph-Based Document Classification: Learning Data-Driven Structures Beyond Heuristic Approachescited by 1
  6. 2026From Global to Local: Learning Context-Aware Graph Representations for Document Classification and Summarizationcited by 0