# BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items

arXiv: https://arxiv.org/abs/2610.13607 · HTML: https://mlsanity.com/paper/2610.13607/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 56 references that are in the map.
- First posted on arXiv 9 Oct 2026.

## Builds on (most cited first, 14 of 14)

- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias](https://mlsanity.com/paper/2404.00929/) (2024)
- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [CLiMP: A Benchmark for Chinese Language Model Evaluation](https://mlsanity.com/paper/2101.11131/) (2021)
- [UniMorph 2.0: Universal Morphology](https://mlsanity.com/paper/1810.11101/) (2018)
- [Language Models Use Monotonicity to Assess NPI Licensing](https://mlsanity.com/paper/2105.13818/) (2021)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languages](https://mlsanity.com/paper/2403.00686/) (2024)
- [RuBLiMP: Russian Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2406.19232/) (2024)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [KoBALT: Korean Benchmark For Advanced Linguistic Tasks](https://mlsanity.com/paper/2505.16125/) (2025)
- [Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models](https://mlsanity.com/paper/2105.04688/) (2021)

## Abstract (arXiv)

Negative polarity items (NPIs), such as "anymore" and "ever", are grammatical only in select environments, predominantly in the context of negation. Because their grammaticality depends on both syntax and semantics, they have a rich, multilingual literature in theoretical linguistics, yet work on NPIs in language models is almost entirely confined to English. We introduce BLiMPPI, a multilingual evaluation suite of over 12,000 NPI minimal pairs in 33 languages, built by a pipeline that discovers NPIs in Universal Dependencies (UD) treebanks, labels their licensing environments, and perturbs these into minimal pairs. Evaluating 12 multilingual and 2 monolingual models alongside n-gram baselines, we find that multilingual models struggle most with lower-resource languages and with pairs that are not n-gram solvable. We further examine the effects of NPI typology, language family, licensing variability, and NPI-licenser distance. Data: https://github.com/arzuburcuguven/blimppi-data