# Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

arXiv: https://arxiv.org/abs/2603.15295 · HTML: https://mlsanity.com/paper/2603.15295/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 56 references that are in the map.
- First posted on arXiv 16 Mar 2026.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?](https://mlsanity.com/paper/2005.00719/) (2020)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs](https://mlsanity.com/paper/2504.02768/) (2025)
- [Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs](https://mlsanity.com/paper/1909.02597/) (2019)
- [Verb Argument Structure Alternations in Word and Sentence Embeddings](https://mlsanity.com/paper/1811.10773/) (2018)
- [Bayes in the age of intelligent machines](https://mlsanity.com/paper/2311.10206/) (2023)
- [A Second Wave of UD Hebrew Treebanking and Cross-Domain Parsing](https://mlsanity.com/paper/2210.07873/) (2022)
- [Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specifications](https://mlsanity.com/paper/2306.11444/) (2023)
- [Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification](https://mlsanity.com/paper/2407.18119/) (2024)
- [Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement](https://mlsanity.com/paper/2409.06567/) (2024)
- [Exploring Italian sentence embeddings properties through multi-tasking](https://mlsanity.com/paper/2409.06622/) (2024)
- [Does Training on Synthetic Data Make Models Less Robust?](https://mlsanity.com/paper/2502.07164/) (2025)
- [Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://mlsanity.com/paper/2602.05648/) (2026)
- [Analogical Structure, Minimal Contextual Cues and Contrastive Distractors: Input Design for Sample-Efficient Linguistic Rule Induction](https://mlsanity.com/paper/2511.10441/) (2025)

## Built on by (most cited first, 1 of 1)

- [Comparing Natural and Synthetic Structured Data: A Study of the Passive Verb Alternation in French and Italian](https://mlsanity.com/paper/2603.25227/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.