# ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

arXiv: https://arxiv.org/abs/2003.10555 · HTML: https://mlsanity.com/paper/2003.10555/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2020.

## Built on by (most cited first, 27 of 27)

- [Neural Approaches to Conversational Information Retrieval](https://mlsanity.com/paper/2201.05176/) (2022)
- [Interactive Natural Language Processing](https://mlsanity.com/paper/2305.13246/) (2023)
- [Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives](https://mlsanity.com/paper/2307.16851/) (2023)
- [One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support](https://mlsanity.com/paper/2306.09237/) (2023)
- [LoRA-LiteE: A Computationally Efficient Framework for Chatbot Preference-Tuning](https://mlsanity.com/paper/2411.09947/) (2024)
- [A Survey of Pre-trained Language Models for Processing Scientific Text](https://mlsanity.com/paper/2401.17824/) (2024)
- [A Survey of Transformer-based Language Models with Focus on Efficiency](https://mlsanity.com/paper/2406.16893/) (2024)
- [Foundation Models for Low-Resource Language Education (Vision Paper)](https://mlsanity.com/paper/2412.04774/) (2024)
- [Enhancing Generalization in Chain of Thought Reasoning for Smaller Models](https://mlsanity.com/paper/2501.09804/) (2025)
- [Danish Foundation Models](https://mlsanity.com/paper/2311.07264/) (2023)
- [KoMultiText: Large-Scale Korean Text Dataset for Classifying Biased Speech in Real-World Online Services](https://mlsanity.com/paper/2310.04313/) (2023)
- [Unfair TOS: An Automated Approach using Customized BERT](https://mlsanity.com/paper/2401.11207/) (2024)
- [Language Modelling Approaches to Adaptive Machine Translation](https://mlsanity.com/paper/2401.14559/) (2024)
- [A Primer on Large Language Models and their Limitations](https://mlsanity.com/paper/2412.04503/) (2024)
- [Case-Grounded Evidence Verification: A Framework for Constructing Evidence-Sensitive Supervision](https://mlsanity.com/paper/2604.09537/) (2026)
- [Fine-Tuning Language Models for Scientific Writing Support](https://mlsanity.com/paper/2306.10974/) (2023)
- [PassionNet: An Innovative Framework for Duplicate and Conflicting Requirements Identification](https://mlsanity.com/paper/2412.01657/) (2024)
- [Cognitive Episodes in LLM Reasoning Traces Enable Interpretable Human Item Difficulty Prediction](https://mlsanity.com/paper/2606.28186/) (2026)
- [STF: Sentence Transformer Fine-Tuning For Topic Categorization With Limited Data](https://mlsanity.com/paper/2407.03253/) (2024)
- [Computational Approaches to Arabic-English Code-Switching](https://mlsanity.com/paper/2410.13318/) (2024)
- [Advancements and Challenges in Bangla Question Answering Models: A Comprehensive Review](https://mlsanity.com/paper/2412.11823/) (2024)
- [Structural Self-Supervised Objectives for Transformers](https://mlsanity.com/paper/2309.08272/) (2023)
- [Teaching Smaller Language Models To Generalise To Unseen Compositional Questions (Full Thesis)](https://mlsanity.com/paper/2411.16985/) (2024)
- [Scaling BERT Models for Turkish Automatic Punctuation and Capitalization Correction](https://mlsanity.com/paper/2412.02698/) (2024)
- [Graph-GRPO: Dependency-Aware Credit Assignment for Generative E-commerce Search Relevance](https://mlsanity.com/paper/2605.31003/) (2026)
- [NoRA: Evaluating Grounded Reasonableness in Visual First-person Normative Action Reasoning](https://mlsanity.com/paper/2606.04806/) (2026)
- [Backdoor Learning in Language Models and Vision-Language Models](https://mlsanity.com/paper/2608.18095/) (2026)

## Abstract (arXiv)

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.