# NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms

arXiv: https://arxiv.org/abs/2402.12261 · HTML: https://mlsanity.com/paper/2402.12261/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 69 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 30 of 34)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation](https://mlsanity.com/paper/2104.14478/) (2021)
- [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://mlsanity.com/paper/2310.03214/) (2023)
- [Mind the Gap: Assessing Temporal Generalization in Neural Language Models](https://mlsanity.com/paper/2102.01951/) (2021)
- [CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task](https://mlsanity.com/paper/2209.06243/) (2022)
- [TimeLMs: Diachronic Language Models from Twitter](https://mlsanity.com/paper/2202.03829/) (2022)
- [TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models](https://mlsanity.com/paper/2204.14211/) (2022)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora](https://mlsanity.com/paper/2110.08534/) (2021)
- [Time Waits for No One! Analysis and Challenges of Temporal Misalignment](https://mlsanity.com/paper/2111.07408/) (2021)
- [Entity Cloze By Date: What LMs Know About Unseen Entities](https://mlsanity.com/paper/2205.02832/) (2022)
- [Federated Learning Of Out-Of-Vocabulary Words](https://mlsanity.com/paper/1903.10635/) (2019)
- [Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media](https://mlsanity.com/paper/2104.08116/) (2021)
- [Temporal Effects on Pre-trained Models for Language Processing Tasks](https://mlsanity.com/paper/2111.12790/) (2021)
- [Few-Shot Representation Learning for Out-Of-Vocabulary Words](https://mlsanity.com/paper/1907.00505/) (2019)
- [Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views](https://mlsanity.com/paper/2302.12297/) (2023)
- [Where New Words Are Born: Distributional Semantic Analysis of Neologisms and Their Semantic Neighborhoods](https://mlsanity.com/paper/2001.07740/) (2020)
- [On the Impact of Temporal Concept Drift on Model Explanations](https://mlsanity.com/paper/2210.09197/) (2022)
- [NYTWIT: A Dataset of Novel Words in the New York Times](https://mlsanity.com/paper/2003.03444/) (2020)
- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [Do CoNLL-2003 Named Entity Taggers Still Work Well in 2023?](https://mlsanity.com/paper/2212.09747/) (2022)
- [The structure of online social networks modulates the rate of lexical change](https://mlsanity.com/paper/2104.05010/) (2021)
- [Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp](https://mlsanity.com/paper/2104.09742/) (2021)
- [A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code](https://mlsanity.com/paper/2010.12663/) (2020)

## Built on by (most cited first, 20 of 20)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [NewTerm: Benchmarking Real-Time New Terms for Large Language Models with Annual Updates](https://mlsanity.com/paper/2410.20814/) (2024)
- [Turkronicles: Diachronic Resources for the Fast Evolving Turkish Language](https://mlsanity.com/paper/2405.10133/) (2024)
- [Same Meaning, Different Scores: Lexical and Syntactic Sensitivity in LLM Evaluation](https://mlsanity.com/paper/2602.17316/) (2026)
- [NeoN: A Tool for Automated Detection, Linguistic and LLM-Driven Analysis of Neologisms in Polish](https://mlsanity.com/paper/2505.15426/) (2025)
- [NeoAMT: Neologism-Aware Agentic Machine Translation with Reinforcement Learning](https://mlsanity.com/paper/2601.03790/) (2026)
- [Benchmarking Machine Translation on Chinese Social Media Texts](https://mlsanity.com/paper/2601.22931/) (2026)
- [Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologisms](https://mlsanity.com/paper/2604.23842/) (2026)
- [Observatorio Lazaro: A self-populating database of anglicism usage in the Spanish press](https://mlsanity.com/paper/2608.00713/) (2026)
- [Detecting Turkish Synonyms Used in Different Time Periods](https://mlsanity.com/paper/2411.15768/) (2024)
- [Hatevolution: What Static Benchmarks Don't Tell Us](https://mlsanity.com/paper/2506.12148/) (2025)
- [Neologism Learning as a Parameter-Efficient Alternative to Fine-Tuning for Model Steering](https://mlsanity.com/paper/2512.18551/) (2025)
- [From sunblock to softblock: Analyzing the correlates of neology in published writing and on social media](https://mlsanity.com/paper/2602.13123/) (2026)
- [OpenFS: Multi-Hand-Capable Fingerspelling Recognition with Implicit Signing-Hand Detection and Frame-Wise Letter-Conditioned Synthesis](https://mlsanity.com/paper/2602.22949/) (2026)
- [Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models](https://mlsanity.com/paper/2605.21227/) (2026)
- [New Terms, New Toxicity: Consensus-based Chinese Neologism Toxicity Detection via Search-Augmented LLMs](https://mlsanity.com/paper/2608.12361/) (2026)
- [CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms](https://mlsanity.com/paper/2608.28053/) (2026)
- [Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs](https://mlsanity.com/paper/2609.00184/) (2026)
- [KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms](https://mlsanity.com/paper/2609.19916/) (2026)
- [KoAGE: Do Large Language Models Know, Learn, and Coin Korean Neologisms?](https://mlsanity.com/paper/2610.19399/) (2026)

## Abstract (arXiv)

The performance of Large Language Models (LLMs) degrades from the temporal drift between data used for model training and newer text seen during inference. One understudied avenue of language change causing data drift is the emergence of neologisms -- new word forms -- over time. We create a diverse resource of recent English neologisms by using several popular collection methods. We analyze temporal drift using neologisms by comparing sentences containing new words with near-identical sentences that replace neologisms with existing substitute words. Model performance is nearly halved in machine translation when a single neologism is introduced in a sentence. Motivated by these results, we construct a benchmark to evaluate LLMs' ability to generalize to neologisms with various natural language understanding tasks and model perplexity. Models with later knowledge cutoff dates yield lower perplexities and perform better in downstream tasks. LLMs are also affected differently based on the linguistic origins of words, indicating that neologisms are complex for static LLMs to address. We will release our benchmark and code for reproducing our experiments.