mlsanityData to 7 Oct 2026

BERT: A Review of Applications in Natural Language Processing and Understanding

arXiv 2103.11943 ↗First posted 22 Mar 2021cs.CL cs.AI cs.LGcited by 109

Not on the map. Why?

Abstract (arXiv)

In this review, we describe the application of one of the most popular deep learning-based language models - BERT. The paper describes the mechanism of operation of this model, the main areas of its application to the tasks of text analytics, comparisons with similar models in each task, as well as a description of some proprietary models. In preparing this review, the data of several dozen original scientific articles published over the past few years, which attracted the most attention in the scientific community, were systematized. This survey will be useful to all students and researchers who want to get acquainted with the latest advances in the field of natural language text analysis.

Key facts. Cited by 109 papers in the map (data through 7 Oct 2026). Builds on 58 of its 99 references that are in the map. First posted on arXiv 22 Mar 2021.

Builds on

58 of 99 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2018Deep contextualized word representationscited by 2,285
38 more
  1. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  2. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  3. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  4. 2016Adversarial examples in the physical worldcited by 1,762
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  6. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  7. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  8. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  11. 2016Practical Black-Box Attacks against Machine Learningcited by 893
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  13. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  14. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  15. 2018Universal Sentence Encodercited by 485
  16. 2017Word Translation Without Parallel Datacited by 480
  17. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  18. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  19. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  20. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  21. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  22. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  23. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  24. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 272
  25. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  26. 2016Understanding Neural Networks through Representation Erasurecited by 223
  27. 2018A Simple Method for Commonsense Reasoningcited by 223
  28. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 212
  29. 2018Scaling Neural Machine Translationcited by 204
  30. 2015Semi-supervised Sequence Learningcited by 198
  31. 2017Unsupervised Neural Machine Translationcited by 148
  32. 2016Recurrent Neural Network for Text Classification with Multi-Task Learningcited by 144
  33. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 138
  34. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 129
  35. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  36. 2017Deep Text Classification Can be Fooledcited by 105
  37. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  38. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 94

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 6; 2023: 10; 2024: 22; 2025: 47; 2026: 23.

109 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 439
  3. 2025IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speechcited by 85
  4. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  5. 2024A Comprehensive Study on Quantization Techniques for Large Language Modelscited by 34
  6. 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
  7. 2025A Survey of Personalization: From RAG to Agentcited by 26
  8. 2024The Landscape and Challenges of HPC Research and LLMscited by 22
  9. 2024MMREC: LLM Based Multi-Modal Recommender Systemcited by 22
  10. 2023Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applicationscited by 21
  11. 2025TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoningcited by 19
  12. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
38 more
  1. 2022Metaethical Perspectives on 'Benchmarking' AI Ethicscited by 11
  2. 2024Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysiscited by 11
  3. 2025Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Modelscited by 10
  4. 2023Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learningcited by 9
  5. 2024An Explainable Transformer-based Model for Phishing Email Detection: A Large Language Model Approachcited by 9
  6. 2025Deep Semantic Graph Learning via LLM based Node Enhancementcited by 8
  7. 2025SpeakStream: Streaming Text-to-Speech with Interleaved Datacited by 8
  8. 2024Rethinking Emotion Annotations in the Era of Large Language Modelscited by 7
  9. 2022ASR-Generated Text for Language Model Pre-training Applied to Speech Taskscited by 6
  10. 2024RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevancecited by 6
  11. 2025Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstructioncited by 6
  12. 2026End-to-End Training for Unified Tokenization and Latent Denoisingcited by 6
  13. 2023Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?cited by 4
  14. 2024TabKANet: Tabular Data Modeling with Kolmogorov-Arnold Network and Transformercited by 4
  15. 2024Optimizing Large Language Models with an Enhanced LoRA Fine-Tuning Algorithm for Efficiency and Robustness in NLP Taskscited by 4
  16. 2025A Genealogy of Foundation Models in Remote Sensingcited by 4
  17. 2025Large Language Models for EEG: A Comprehensive Survey and Taxonomycited by 4
  18. 2025Act-With-Think: Chunk Auto-Regressive Modeling for Generative Recommendationcited by 4
  19. 2025Identify, Isolate, and Purge: Mitigating Hallucinations in LVLMs via Self-Evolving Distillationcited by 4
  20. 2025MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operationscited by 4
  21. 2026Are LLM-Enhanced Graph Neural Networks Robust against Poisoning Attacks?cited by 4
  22. 2022Finding Reusable Machine Learning Components to Build Programming Language Processing Pipelinescited by 3
  23. 2024SpamDam: Towards Privacy-Preserving and Adversary-Resistant SMS Spam Detectioncited by 3
  24. 2024Revolutionizing Bridge Operation and Maintenance with LLM-based Agents: An Overview of Applications and Insightscited by 3
  25. 2025Leveraging Online Data to Enhance Medical Knowledge in a Small Persian Language Modelcited by 3
  26. 2025KEA Explain: Explanations of Hallucinations using Graph Kernel Analysiscited by 3
  27. 2025Can LLMs Find Bugs in Code? An Evaluation from Beginner Errors to Security Vulnerabilities in Python and C++cited by 3
  28. 2025StockMem: An Event-Reflection Memory Framework for Stock Forecastingcited by 3
  29. 2025MSME: A Multi-Stage Multi-Expert Framework for Zero-Shot Stance Detectioncited by 3
  30. 2025ChipMind: Retrieval-Augmented Reasoning for Long-Context Circuit Design Specificationscited by 3
  31. 2025Fast SAM2 with Text-Driven Token Pruningcited by 3
  32. 2026Linguistics and Human Brain: A Perspective of Computational Neurosciencecited by 3
  33. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priorscited by 3
  34. 2026FactGuard: Agentic Video Misinformation Detection via Reinforcement Learningcited by 3
  35. 2021Literature review on vulnerability detection using NLP technologycited by 2
  36. 2023"I'm Not Confident in Debiasing AI Systems Since I Know Too Little": Teaching AI Creators About Gender Bias Through Hands-on Tutorialscited by 2
  37. 2024ExplainableDetector: Exploring Transformer-based Language Modeling Approach for SMS Spam Detection with Explainability Analysiscited by 2
  38. 2024EUREKHA: Enhancing User Representation for Key Hackers Identification in Underground Forumscited by 2

50 most cited of 109 All 109 (JSON)