# Spark NLP: Natural Language Understanding at Scale

arXiv: https://arxiv.org/abs/2101.10848 · HTML: https://mlsanity.com/paper/2101.10848/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 26 Jan 2021.

## Builds on (most cited first, 8 of 8)

- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [A Survey on Recent Advances in Named Entity Recognition from Deep Learning models](https://mlsanity.com/paper/1910.11470/) (2019)
- [Biomedical Named Entity Recognition at Scale](https://mlsanity.com/paper/2011.06315/) (2020)
- [Cross-type Biomedical Named Entity Recognition with Deep Multi-Task Learning](https://mlsanity.com/paper/1801.09851/) (2018)
- [Improving Clinical Document Understanding on COVID-19 Research with Spark NLP](https://mlsanity.com/paper/2012.04005/) (2020)
- [Biomedical and Clinical English Model Packages in the Stanza Python NLP Library](https://mlsanity.com/paper/2007.14640/) (2020)

## Built on by (most cited first, 16 of 16)

- [MedPerf: Open Benchmarking Platform for Medical Artificial Intelligence using Federated Evaluation](https://mlsanity.com/paper/2110.01406/) (2021)
- [Deeper Clinical Document Understanding Using Relation Extraction](https://mlsanity.com/paper/2112.13259/) (2021)
- [Regularization for Long Named Entity Recognition](https://mlsanity.com/paper/2104.07249/) (2021)
- [Large language models for newspaper sentiment analysis during COVID-19: The Guardian](https://mlsanity.com/paper/2405.13056/) (2024)
- [Large-Scale News Classification using BERT Language Model: Spark NLP Approach](https://mlsanity.com/paper/2107.06785/) (2021)
- [Beyond Accuracy: Automated De-Identification of Large Real-World Clinical Text Datasets](https://mlsanity.com/paper/2312.08495/) (2023)
- [Understanding COVID-19 News Coverage using Medical NLP](https://mlsanity.com/paper/2203.10338/) (2022)
- [Bridging the Domain Divide: Supervised vs. Zero-Shot Clinical Section Segmentation from MIMIC-III to Obstetrics](https://mlsanity.com/paper/2602.17513/) (2026)
- [An Evaluation of Persian-English Machine Translation Datasets with Transformers](https://mlsanity.com/paper/2302.00321/) (2023)
- [Extraction of Medication and Temporal Relation from Clinical Text using Neural Language Models](https://mlsanity.com/paper/2310.02229/) (2023)
- [KhabarChin: Automatic Detection of Important News in the Persian Language](https://mlsanity.com/paper/2312.03361/) (2023)
- [Beyond Negation Detection: Comprehensive Assertion Detection Models for Clinical NLP](https://mlsanity.com/paper/2503.17425/) (2025)
- [Can Zero-Shot Commercial APIs Deliver Regulatory-Grade Clinical Text DeIdentification?](https://mlsanity.com/paper/2503.20794/) (2025)
- [A Biomedical Pipeline to Detect Clinical and Non-Clinical Named Entities](https://mlsanity.com/paper/2207.00876/) (2022)
- [Implicit Framing in Obstetric Counseling Notes: A Grounded LLM Pipeline on a VBAC-Eligible Cohort](https://mlsanity.com/paper/2604.23059/) (2026)
- [Building Socio-Affective Artificial Intelligence for Interactive Multi-Agent Simulations](https://mlsanity.com/paper/2609.26927/) (2026)

## Abstract (arXiv)

Spark NLP is a Natural Language Processing (NLP) library built on top of Apache Spark ML. It provides simple, performant and accurate NLP annotations for machine learning pipelines that can scale easily in a distributed environment. Spark NLP comes with 1100 pre trained pipelines and models in more than 192 languages. It supports nearly all the NLP tasks and modules that can be used seamlessly in a cluster. Downloaded more than 2.7 million times and experiencing nine times growth since January 2020, Spark NLP is used by 54% of healthcare organizations as the worlds most widely used NLP library in the enterprise.