mlsanityData to 7 Oct 2026

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

arXiv 2006.03654 ↗First posted 5 Jun 2020cs.CL cs.LGcited by 2,157

Not on the map. Why?

Cited by 96 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline by a decent margin (90.3 versus 89.8).

Key facts. Cited by 2,157 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 164; 2022: 284; 2023: 426; 2024: 523; 2025: 461; 2026: 299.

2,157 papers.

Sort:

By area:

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  12. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  4. 2024Large Language Models: A Surveycited by 506
  5. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  7. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  8. 2021A Survey of Transformerscited by 399
  9. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
  13. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  14. 2023Explainability for Large Language Models: A Surveycited by 331
  15. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  16. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  17. 2022UL2: Unifying Language Learning Paradigmscited by 292
  18. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  19. 2025Memory in the Age of AI Agentscited by 285
  20. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 283
  21. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  22. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  23. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 264
  24. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  25. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  26. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  27. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  28. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  29. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  30. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  31. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 228
  32. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  33. 2021WARP: Word-level Adversarial ReProgrammingcited by 220
  34. 2023On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspectivecited by 208
  35. 2021ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Predictioncited by 204
  36. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  37. 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 200
  38. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199

50 most cited of 2,157 All 2,157 (JSON)