mlsanityData to 7 Oct 2026

Pre-Training Transformers as Energy-Based Cloze Models

arXiv 2012.08561 ↗First posted 15 Dec 2020cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

We introduce Electric, an energy-based cloze model for representation learning over text. Like BERT, it is a conditional generative model of tokens given their contexts. However, Electric does not use masking or output a full distribution over tokens that could occur in a context. Instead, it assigns a scalar energy score to each input token indicating how likely it is given its context. We train Electric using an algorithm based on noise-contrastive estimation and elucidate how this learning objective is closely related to the recently proposed ELECTRA pre-training method. Electric performs well when transferred to downstream tasks and is particularly effective at producing likelihood scores for text: it re-ranks speech recognition n-best lists better than language models and much faster than masked language models. Furthermore, it offers a clearer and more principled view of what ELECTRA learns during pre-training.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 12; 2023: 13; 2024: 5; 2025: 2; 2026: 1.

52 papers.

Sort:

By area:

  1. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  3. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  4. 2021Step-unrolled Denoising Autoencoders for Text Generationcited by 102
  5. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  6. 2021How to Train BERT with an Academic Budgetcited by 80
  7. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  8. 2022OpenCQA: Open-ended Question Answering with Chartscited by 60
  9. 2021Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Mediacited by 51
  10. 2021A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectivescited by 41
  11. 2021FewCLUE: A Chinese Few-shot Learning Evaluation Benchmarkcited by 41
  12. 2021Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastingscited by 37
38 more
  1. 2023Defending against Insertion-based Textual Backdoor Attacks via Attributioncited by 24
  2. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  3. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  4. 2022Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddingscited by 18
  5. 2021Frustratingly Simple Pretraining Alternatives to Masked Language Modelingcited by 17
  6. 2023Interpretable Math Word Problem Solution Generation Via Step-by-step Planningcited by 17
  7. 2021ASR Rescoring and Confidence Estimation with ELECTRAcited by 16
  8. 2023Refashioning Emotion Recognition Modelling: The Advent of Generalised Large Modelscited by 16
  9. 2022Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistencycited by 11
  10. 2022Transcormer: Transformer for Sentence Scoring with Sliding Language Modelingcited by 8
  11. 2021Unified Interpretation of Softmax Cross-Entropy and Negative Sampling: With Case Study for Knowledge Graph Embeddingcited by 7
  12. 2021Learning to Sample Replacements for ELECTRA Pre-Trainingcited by 7
  13. 2022SMTCE: A Social Media Text Classification Evaluation Benchmark and BERTology Models for Vietnamesecited by 7
  14. 2021Contrastive Document Representation Learning with Graph Attention Networkscited by 6
  15. 2022Representation Learning by Detecting Incorrect Location Embeddingscited by 5
  16. 2023Pseudo Outlier Exposure for Out-of-Distribution Detection using Pretrained Transformerscited by 5
  17. 2023Headless Language Models: Learning without Predicting with Contrastive Weight Tyingcited by 5
  18. 2021Training ELECTRA Augmented with Multi-word Selectioncited by 4
  19. 2022On Measuring the Intrinsic Few-Shot Hardness of Datasetscited by 4
  20. 2021Lacuna Reconstruction: Self-supervised Pre-training for Low-Resource Historical Document Transcriptioncited by 3
  21. 2023Subtle Misogyny Detection and Mitigation: An Expert-Annotated Datasetcited by 3
  22. 2023LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spacescited by 3
  23. 2024MolMix: A Simple Yet Effective Baseline for Multimodal Molecular Representation Learningcited by 3
  24. 2024Transformers Utilization in Chart Understanding: A Review of Recent Advances & Future Trendscited by 3
  25. 2022PATS: Sensitivity-aware Noisy Learning for Pretrained Language Modelscited by 2
  26. 2023Exploiting Abstract Meaning Representation for Open-Domain Question Answeringcited by 2
  27. 2023Controlled Text Generation for Black-box Language Models via Score-based Progressive Editorcited by 2
  28. 2021SAS: Self-Augmentation Strategy for Language Model Pre-trainingcited by 1
  29. 2022Unsupervised Term Extraction for Highly Technical Domainscited by 1
  30. 2023Exploring Energy-based Language Models with Different Architectures and Training Methods for Speech Recognitioncited by 1
  31. 2023Energy-Based Models For Speech Synthesiscited by 1
  32. 2025ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMscited by 1
  33. 2026ClaimFlow: Tracing the Evolution of Scientific Claims in NLPcited by 1
  34. 2021A Game Interface to Study Semantic Grounding in Text-Based Modelscited by 0
  35. 2021Unsupervised Open-Domain Question Answeringcited by 0
  36. 2022Robustifying Sentiment Classification by Maximally Exploiting Few Counterfactualscited by 0
  37. 2024Exploring Internal Numeracy in Language Models: A Case Study on ALBERTcited by 0
  38. 2024Entangled Relations: Leveraging NLI and Meta-analysis to Enhance Biomedical Relation Extractioncited by 0

50 most cited of 52 All 52 (JSON)