mlsanityData to 14 Oct 2026

Efficient transfer learning for NLP with ELECTRA

arXiv 2104.02756 ↗First posted 6 Apr 2021cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Clark et al. [2020] claims that the ELECTRA approach is highly efficient in NLP performances relative to computation budget. As such, this reproducibility study focus on this claim, summarized by the following question: Can we use ELECTRA to achieve close to SOTA performances for NLP in low-resource settings, in term of compute cost?

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 11 of its 21 references that are in the map. First posted on arXiv 6 Apr 2021.

Builds on

11 of 21 references.

References · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2018Deep contextualized word representationscited by 2,290
  8. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  11. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215

Built on by

Not cited yet.