mlsanityData to 7 Oct 2026

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

arXiv 2111.09543 ↗First posted 18 Nov 2021cs.CL cs.LGcited by 1,219

Not on the map. Why?

Cited by 56 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the "tug-of-war" dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

Key facts. Cited by 1,219 papers in the map (data through 7 Oct 2026). Builds on 41 of its 65 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

41 of 65 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
29 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  8. 2017Decoupled Weight Decay Regularizationcited by 1,722
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  13. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  14. 2018Self-Attention with Relative Position Representationscited by 1,112
  15. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  16. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  17. 2018Neural Network Acceptability Judgmentscited by 952
  18. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  19. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  20. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  21. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  22. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  23. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  24. 2018A Simple Method for Commonsense Reasoningcited by 223
  25. 2018Music Transformercited by 207
  26. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  27. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  28. 2021XLM-E: Cross-lingual Language Model Pre-training via ELECTRAcited by 72
  29. 2021Small-Bench NLP: Benchmark for small single GPU trained models in Natural Language Processingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 87; 2023: 203; 2024: 321; 2025: 320; 2026: 285.

1,219 papers.

Sort:

By area:

  1. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  2. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  3. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  4. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 396
  5. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  6. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  7. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  8. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  9. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 259
  10. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  11. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  12. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
38 more
  1. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  2. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  3. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
  4. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  5. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  6. 2023Fact-Checking Complex Claims with Program-Guided Reasoningcited by 146
  7. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  8. 2023LingoQA: Visual Question Answering for Autonomous Drivingcited by 144
  9. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  10. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  11. 2023NL4Opt Competition: Formulating Optimization Problems Based on Their Natural Language Descriptionscited by 133
  12. 2023ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigationcited by 126
  13. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  14. 2023GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extractioncited by 116
  15. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 115
  16. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  17. 2022Latent Diffusion for Language Generationcited by 110
  18. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  19. 2023Mustango: Toward Controllable Text-to-Music Generationcited by 100
  20. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  21. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  22. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  23. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  24. 2023A Survey on Fairness in Large Language Modelscited by 91
  25. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  26. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 84
  27. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  28. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  29. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  30. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  31. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  32. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  33. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  34. 2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
  35. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  36. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  37. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  38. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65

50 most cited of 1,219 All 1,219 (JSON)