mlsanityData to 14 Oct 2026

Headless Language Models: Learning without Predicting with Contrastive Weight Tying

arXiv 2309.08351 ↗First posted 15 Sep 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative method that shifts away from probability prediction and instead focuses on reconstructing input embeddings in a contrastive fashion via Constrastive Weight Tying (CWT). We apply this approach to pretrain Headless Language Models in both monolingual and multilingual contexts. Our method offers practical advantages, substantially reducing training computational requirements by up to 20 times, while simultaneously enhancing downstream performance and data efficiency. We observe a significant +1.6 GLUE score increase and a notable +2.7 LAMBADA accuracy improvement compared to classical LMs within similar compute budgets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 15 Sep 2023.

Builds on

33 of 46 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  10. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  12. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
21 more
  1. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  2. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  3. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  4. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  6. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  7. 2016Using the Output Embedding to Improve Language Modelscited by 272
  8. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 255
  9. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  10. 2022A Contrastive Framework for Neural Text Generationcited by 225
  11. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  12. 2023Tokenization and the Noiseless Channelcited by 59
  13. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  14. 2021Investigating the Role of Negatives in Contrastive Representation Learningcited by 54
  15. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  16. 2022Do More Negative Samples Necessarily Hurt in Contrastive Learning?cited by 39
  17. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  18. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  19. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 20
  20. 2022ContraCLM: Contrastive Learning For Causal Language Modelcited by 18
  21. 2022miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddingscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 24
  2. 2025Gaperon: A Peppered English-French Generative Language Model Suitecited by 9
  3. 2024Zipfian Whiteningcited by 3
  4. 2025SpeLLM: Character-Level Multi-Head Decodingcited by 2
  5. 2026Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Losscited by 0