mlsanityData to 7 Oct 2026

HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization

arXiv 1905.06566 ↗First posted 16 May 2019cs.CL cs.LGcited by 120

Not on the map. Why?

Abstract (arXiv)

Neural extractive summarization models usually employ a hierarchical encoder for document encoding and they are trained using sentence-level labels, which are created heuristically using rule-based methods. Training the hierarchical encoder with these inaccurate labels is challenging. Inspired by the recent work on pre-training transformer sentence encoders citedevlin:2018:arxiv, we propose sc Hibert (as shorthand for bf HIerachical bf Bidirectional bf Encoder bf Representations from bf Transformers) for document encoding and a method to pre-train it using unlabeled data. We apply the pre-trained sc Hibert to our summarization model and it outperforms its randomly initialized counterpart by 1.25 ROUGE on the CNN/Dailymail dataset and by 2.0 ROUGE on a version of New York Times dataset. We also achieve the state-of-the-art performance on these two datasets.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 41; 2022: 38; 2023: 17; 2024: 8; 2025: 12; 2026: 4.

120 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 399
  2. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  5. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  6. 2021Nested Hierarchical Transformer: Towards Accurate, Data-Efficient and Interpretable Visual Understandingcited by 97
  7. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  8. 2023Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Methodcited by 76
  9. 2021Making Transformers Solve Compositional Taskscited by 72
  10. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  11. 2024ThinK: Thinner Key Cache by Query-Driven Pruningcited by 66
  12. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
38 more
  1. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
  2. 2023A Survey on Long Text Modeling with Transformerscited by 47
  3. 2021Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherencecited by 45
  4. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  5. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  6. 2021Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modelingcited by 35
  7. 2021Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learningcited by 28
  8. 2022Toward Unifying Text Segmentation and Long Document Summarizationcited by 28
  9. 2021Controllable Abstractive Dialogue Summarization with Sketch Supervisioncited by 27
  10. 2021VILA: Improving Structured Content Extraction from Scientific PDFs Using Visual Layout Groupscited by 27
  11. 2022HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Informationcited by 26
  12. 2023DiffuSum: Generation Enhanced Extractive Summarization with Diffusioncited by 25
  13. 2023RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generationcited by 25
  14. 2021Long-Span Summarization via Local Attention and Content Selectioncited by 24
  15. 2021Topic-Aware Contrastive Learning for Abstractive Dialogue Summarizationcited by 23
  16. 2022HIBRIDS: Attention with Hierarchical Biases for Structure-aware Long Document Summarizationcited by 22
  17. 2021Enriching and Controlling Global Semantics for Text Summarizationcited by 20
  18. 2021MemSum: Extractive Summarization of Long Documents Using Multi-Step Episodic Markov Decision Processescited by 19
  19. 2021Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretrainingcited by 19
  20. 2021Topic-Guided Abstractive Multi-Document Summarizationcited by 19
  21. 2021CDLM: Cross-Document Language Modelingcited by 17
  22. 2022RoSGAS: Adaptive Social Bot Detection with Reinforced Self-Supervised GNN Architecture Searchcited by 17
  23. 2022LSG Attention: Extrapolation of pretrained Transformers to long sequencescited by 16
  24. 2021Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summariescited by 15
  25. 2021ReadTwice: Reading Very Large Documents with Memoriescited by 14
  26. 2021Leveraging Information Bottleneck for Scientific Document Summarizationcited by 13
  27. 2021HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarizationcited by 13
  28. 2022An Automatic and Efficient BERT Pruning for Edge AI Systemscited by 13
  29. 2023MMSum: A Dataset for Multimodal Summarization and Thumbnail Generation of Videoscited by 12
  30. 2021MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detectioncited by 11
  31. 2023Linear Log-Normal Attention with Unbiased Concentrationcited by 11
  32. 2025Time Series Language Model for Descriptive Caption Generationcited by 11
  33. 2021Predicting Discourse Trees from Transformer-based Neural Summarizerscited by 10
  34. 2021Iterative Hierarchical Attention for Answering Complex Questions over Long Documentscited by 10
  35. 2021Code-switched inspired losses for generic spoken dialog representationscited by 10
  36. 2022GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarizationcited by 10
  37. 2024Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applicationscited by 10
  38. 2022Read Top News First: A Document Reordering Approach for Multi-Document News Summarizationcited by 9

50 most cited of 120 All 120 (JSON)