mlsanityData to 14 Oct 2026

Better Language Model with Hypernym Class Prediction

arXiv 2203.10692 ↗First posted 21 Mar 2022cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Class-based language models (LMs) have been long devised to address context sparsity in n-gram LMs. In this study, we revisit this approach in the context of neural LMs. We hypothesize that class-based prediction leads to an implicit context aggregation for similar words and thus can improve generalization for rare words. We map words that have a common WordNet hypernym to the same class and train large neural LMs by gradually annealing from predicting the class to token prediction during training. Empirically, this curriculum learning strategy consistently improves perplexity over various large, highly-performant state-of-the-art Transformer-based models on two datasets, WikiText-103 and Arxiv. Our analysis shows that the performance improvement is achieved without sacrificing performance on rare words. Finally, we document other attempts that failed to yield empirical gains, and discuss future directions for the adoption of class-based LMs on a larger scale.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 26 of its 40 references that are in the map. First posted on arXiv 21 Mar 2022.

Builds on

26 of 40 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Longformer: The Long-Document Transformercited by 3,413
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  11. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  12. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
14 more
  1. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
  2. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  3. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  4. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  5. 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
  6. 2020When Do Curricula Work?cited by 106
  7. 2020Residual Energy-Based Models for Text Generationcited by 95
  8. 2016Improving Neural Language Models with a Continuous Cachecited by 82
  9. 2020Shortformer: Better Language Modeling using Shorter Inputscited by 76
  10. 2019SenseBERT: Driving Some Sense into BERTcited by 73
  11. 2016Efficient softmax approximation for GPUscited by 67
  12. 2019Latent Normalizing Flows for Discrete Sequencescited by 67
  13. 2021Modeling Event Plausibility with Consistent Conceptual Abstractioncited by 20
  14. 2019Enriching Rare Word Representations in Neural Language Models by Embedding Matrix Augmentationcited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 1.

5 papers.

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  2. 2023CLIMB: Curriculum Learning for Infant-inspired Model Buildingcited by 20
  3. 2023On the Impact of Cross-Domain Data on German Language Modelscited by 5
  4. 2025Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratingscited by 3
  5. 2024Talking the Talk Does Not Entail Walking the Walk: On the Limits of Large Language Models in Lexical Entailment Recognitioncited by 1