mlsanityData to 7 Oct 2026

Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

arXiv 2204.03243 ↗First posted 7 Apr 2022cs.CL cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

We present a new framework AMOS that pretrains text encoders with an Adversarial learning curriculum via a Mixture Of Signals from multiple auxiliary generators. Following ELECTRA-style pretraining, the main encoder is trained as a discriminator to detect replaced tokens generated by auxiliary masked language models (MLMs). Different from ELECTRA which trains one MLM as the generator, we jointly train multiple MLMs of different sizes to provide training signals at various levels of difficulty. To push the discriminator to learn better with challenging replaced tokens, we learn mixture weights over the auxiliary MLMs' outputs to maximize the discriminator loss by backpropagating the gradient from the discriminator via Gumbel-Softmax. For better pretraining efficiency, we propose a way to assemble multiple MLMs into one unified auxiliary model. AMOS outperforms ELECTRA and recent state-of-the-art pretrained models by about 1 point on the GLUE benchmark for BERT base-sized models.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 7 Apr 2022.

Builds on

37 of 58 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2022Generative Adversarial Networkscited by 5,603
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
25 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2018Neural Network Acceptability Judgmentscited by 952
  7. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  8. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  10. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  11. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  12. 2017Automated Curriculum Learning for Neural Networkscited by 251
  13. 2018A Simple Method for Commonsense Reasoningcited by 223
  14. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  15. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  16. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  17. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  18. 2018Language GANs Falling Shortcited by 97
  19. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  20. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  21. 2017Adversarial Feature Matching for Text Generationcited by 58
  22. 2021Question Answering Infused Pre-training of General-Purpose Contextualized Representationscited by 19
  23. 2020MC-BERT: Efficient Language Pre-Training via a Meta Controllercited by 14
  24. 2018Neural-Davidsonian Semantic Proto-role Labelingcited by 9
  25. 2021Learning to Sample Replacements for ELECTRA Pre-Trainingcited by 7

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3; 2024: 2; 2025: 1.

9 papers.

  1. 2024MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Modelscited by 96
  2. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  3. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20
  4. 2023Representation Deficiency in Masked Language Modelingcited by 10
  5. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  6. 2022Language Model Pre-training on True Negativescited by 1
  7. 2023Model-Generated Pretraining Signals Improves Zero-Shot Generalization of Text-to-Text Transformerscited by 1
  8. 2023Fast-ELECTRA for Efficient Pre-trainingcited by 1
  9. 2024SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detectioncited by 0