# BERT: A Review of Applications in Natural Language Processing and Understanding

arXiv: https://arxiv.org/abs/2103.11943 · HTML: https://mlsanity.com/paper/2103.11943/

## Key facts

- Cited by 109 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 99 references that are in the map.
- First posted on arXiv 22 Mar 2021.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)

## Built on by (most cited first, 30 of 109)

- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2506.21619/) (2025)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [A Comprehensive Study on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2411.02530/) (2024)
- [Large Language Models for UAVs: Current State and Pathways to the Future](https://mlsanity.com/paper/2405.01745/) (2024)
- [A Survey of Personalization: From RAG to Agent](https://mlsanity.com/paper/2504.10147/) (2025)
- [The Landscape and Challenges of HPC Research and LLMs](https://mlsanity.com/paper/2402.02018/) (2024)
- [MMREC: LLM Based Multi-Modal Recommender System](https://mlsanity.com/paper/2408.04211/) (2024)
- [Exploring New Frontiers in Agricultural NLP: Investigating the Potential of Large Language Models for Food Applications](https://mlsanity.com/paper/2306.11892/) (2023)
- [TableMind: An Autonomous Programmatic Agent for Tool-Augmented Table Reasoning](https://mlsanity.com/paper/2509.06278/) (2025)
- [Metaethical Perspectives on 'Benchmarking' AI Ethics](https://mlsanity.com/paper/2204.05151/) (2022)
- [Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysis](https://mlsanity.com/paper/2402.09151/) (2024)
- [COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation](https://mlsanity.com/paper/2502.02589/) (2025)
- [Native Visual Understanding: Resolving Resolution Dilemmas in Vision-Language Models](https://mlsanity.com/paper/2506.12776/) (2025)
- [Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning](https://mlsanity.com/paper/2305.08473/) (2023)
- [An Explainable Transformer-based Model for Phishing Email Detection: A Large Language Model Approach](https://mlsanity.com/paper/2402.13871/) (2024)
- [Rethinking Emotion Annotations in the Era of Large Language Models](https://mlsanity.com/paper/2412.07906/) (2024)
- [Deep Semantic Graph Learning via LLM based Node Enhancement](https://mlsanity.com/paper/2502.07982/) (2025)
- [SpeakStream: Streaming Text-to-Speech with Interleaved Data](https://mlsanity.com/paper/2505.19206/) (2025)
- [RelevAI-Reviewer: A Benchmark on AI Reviewers for Survey Paper Relevance](https://mlsanity.com/paper/2406.10294/) (2024)
- [Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction](https://mlsanity.com/paper/2510.14319/) (2025)
- [ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks](https://mlsanity.com/paper/2207.01893/) (2022)
- [End-to-End Training for Unified Tokenization and Latent Denoising](https://mlsanity.com/paper/2603.22283/) (2026)
- [Large Language Models for EEG: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2506.06353/) (2025)
- [FactGuard: Agentic Video Misinformation Detection via Reinforcement Learning](https://mlsanity.com/paper/2602.22963/) (2026)
- [Large Language Models in Cryptocurrency Securities Cases: Can a GPT Model Meaningfully Assist Lawyers?](https://mlsanity.com/paper/2308.06032/) (2023)
- [TabKANet: Tabular Data Modeling with Kolmogorov-Arnold Network and Transformer](https://mlsanity.com/paper/2409.08806/) (2024)
- [Optimizing Large Language Models with an Enhanced LoRA Fine-Tuning Algorithm for Efficiency and Robustness in NLP Tasks](https://mlsanity.com/paper/2412.18729/) (2024)
- [A Genealogy of Foundation Models in Remote Sensing](https://mlsanity.com/paper/2504.17177/) (2025)

## Abstract (arXiv)

In this review, we describe the application of one of the most popular deep learning-based language models - BERT. The paper describes the mechanism of operation of this model, the main areas of its application to the tasks of text analytics, comparisons with similar models in each task, as well as a description of some proprietary models. In preparing this review, the data of several dozen original scientific articles published over the past few years, which attracted the most attention in the scientific community, were systematized. This survey will be useful to all students and researchers who want to get acquainted with the latest advances in the field of natural language text analysis.