NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents
Not on the map. Why?
Abstract (arXiv)
While (large) language models have significantly improved over the last years, they still struggle to sensibly process long sequences found, e.g., in books, due to the quadratic scaling of the underlying attention mechanism. To address this, we propose NextLevelBERT, a Masked Language Model operating not on tokens, but on higher-level semantic representations in the form of text embeddings. We pretrain NextLevelBERT to predict the vector representation of entire masked text chunks and evaluate the effectiveness of the resulting document vectors on three types of tasks: 1) Semantic Textual Similarity via zero-shot document embeddings, 2) Long document classification, 3) Multiple-choice question answering. We find that next-level Masked Language Modeling is an effective technique to tackle long-document use cases and can outperfor much larger embedding models as long as the required level of detail of semantic information is not too fine. Our models and code are publicly available online.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 27 Feb 2024.
Builds on
32 of 43 references.
References · 32
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2020Reformer: The Efficient Transformercited by 1,658
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
20 more
- 2020Mastering Atari with Discrete World Modelscited by 990
- 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
- 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
- 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
- 2022Autoregressive Image Generation using Residual Quantizationcited by 706
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
- 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
- 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
- 2022Should You Mask 15% in Masked Language Modeling?cited by 117
- 2019Hierarchical Transformers for Long Document Classificationcited by 103
- 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
- 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
- 2021Poolingformer: Long Document Modeling with Pooling Attentioncited by 60
- 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
- 2022HEGEL: Hypergraph Transformer for Long Document Summarizationcited by 28
- 2022HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Informationcited by 26
- 2020Unsupervised Extractive Summarization by Pre-training Hierarchical Transformerscited by 16
- 2022MuLD: The Multitask Long Document Benchmarkcited by 9
- 2023Zero and Few-Shot Localization of Task-Oriented Dialogue Agents with a Distilled Representationcited by 4
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.