# Verbing Weirds Language (Models): Evaluation of English Zero-Derivation in Five LLMs

arXiv: https://arxiv.org/abs/2403.17856 · HTML: https://mlsanity.com/paper/2403.17856/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 22 references that are in the map.
- First posted on arXiv 26 Mar 2024.

## Builds on (most cited first, 4 of 4)

- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Words](https://mlsanity.com/paper/2101.00403/) (2021)
- [Where New Words Are Born: Distributional Semantic Analysis of Neologisms and Their Semantic Neighborhoods](https://mlsanity.com/paper/2001.07740/) (2020)
- [Quantifying Cognitive Factors in Lexical Decline](https://mlsanity.com/paper/2110.05775/) (2021)

## Built on by (most cited first, 5 of 5)

- [Robust Privacy Amidst Innovation with Large Language Models Through a Critical Assessment of the Risks](https://mlsanity.com/paper/2407.16166/) (2024)
- [Programming by Examples Meets Historical Linguistics: A Large Language Model Based Approach to Sound Law Induction](https://mlsanity.com/paper/2501.16524/) (2025)
- [Can Large Language Models Code Like a Linguist?: A Case Study in Low Resource Sound Law Induction](https://mlsanity.com/paper/2406.12725/) (2024)
- [From sunblock to softblock: Analyzing the correlates of neology in published writing and on social media](https://mlsanity.com/paper/2602.13123/) (2026)
- [Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions](https://mlsanity.com/paper/2605.31586/) (2026)

## Abstract (arXiv)

Lexical-syntactic flexibility, in the form of conversion (or zero-derivation) is a hallmark of English morphology. In conversion, a word with one part of speech is placed in a non-prototypical context, where it is coerced to behave as if it had a different part of speech. However, while this process affects a large part of the English lexicon, little work has been done to establish the degree to which language models capture this type of generalization. This paper reports the first study on the behavior of large language models with reference to conversion. We design a task for testing lexical-syntactic flexibility -- the degree to which models can generalize over words in a construction with a non-prototypical part of speech. This task is situated within a natural language inference paradigm. We test the abilities of five language models -- two proprietary models (GPT-3.5 and GPT-4), three open-source models (Mistral 7B, Falcon 40B, and Llama 2 70B). We find that GPT-4 performs best on the task, followed by GPT-3.5, but that the open source language models are also able to perform it and that the 7B parameter Mistral displays as little difference between its baseline performance on the natural language inference task and the non-prototypical syntactic category task, as the massive GPT-4.