# MC-BERT: Efficient Language Pre-Training via a Meta Controller

arXiv: https://arxiv.org/abs/2006.05744 · HTML: https://mlsanity.com/paper/2006.05744/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2020.

## Built on by (most cited first, 15 of 15)

- [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://mlsanity.com/paper/2102.08473/) (2021)
- [MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models](https://mlsanity.com/paper/2406.06046/) (2024)
- [METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals](https://mlsanity.com/paper/2204.06644/) (2022)
- [Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators](https://mlsanity.com/paper/2204.03243/) (2022)
- [Building Chinese Biomedical Language Models via Multi-Level Text Discrimination](https://mlsanity.com/paper/2110.07244/) (2021)
- [Training ELECTRA Augmented with Multi-word Selection](https://mlsanity.com/paper/2106.00139/) (2021)
- [SAS: Self-Augmentation Strategy for Language Model Pre-training](https://mlsanity.com/paper/2106.07176/) (2021)
- [Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformers](https://mlsanity.com/paper/2305.12567/) (2023)
- [Fast-ELECTRA for Efficient Pre-training](https://mlsanity.com/paper/2310.07347/) (2023)
- [MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports](https://mlsanity.com/paper/2605.03103/) (2026)
- [Pre-training Language Model as a Multi-perspective Course Learner](https://mlsanity.com/paper/2305.03981/) (2023)
- [Enhancing Context Through Contrast](https://mlsanity.com/paper/2401.03314/) (2024)
- [Enriched text-guided variational multimodal knowledge distillation network (VMD) for automated diagnosis of plaque vulnerability in 3D carotid artery MRI](https://mlsanity.com/paper/2509.11924/) (2025)
- [Atom-level Protein Representation Learning Improves Protein Structure Prediction](https://mlsanity.com/paper/2605.22133/) (2026)
- [UltraBERT: Beyond the Limit of BERT Pre-Training](https://mlsanity.com/paper/2610.15028/) (2026)

## Abstract (arXiv)

Pre-trained contextual representations (e.g., BERT) have become the foundation to achieve state-of-the-art results on many NLP tasks. However, large-scale pre-training is computationally expensive. ELECTRA, an early attempt to accelerate pre-training, trains a discriminative model that predicts whether each input token was replaced by a generator. Our studies reveal that ELECTRA's success is mainly due to its reduced complexity of the pre-training task: the binary classification (replaced token detection) is more efficient to learn than the generation task (masked language modeling). However, such a simplified task is less semantically informative. To achieve better efficiency and effectiveness, we propose a novel meta-learning framework, MC-BERT. The pre-training task is a multi-choice cloze test with a reject option, where a meta controller network provides training input and candidates. Results over GLUE natural language understanding benchmark demonstrate that our proposed method is both efficient and effective: it outperforms baselines on GLUE semantic tasks given the same computational budget.