Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have shown remarkable performance across various sentence-based linguistic phenomena, yet their ability to capture cross-sentence paradigmatic patterns, such as verb alternations, remains underexplored. In this work, we present curated paradigm-based datasets for four languages, designed to probe systematic cross-sentence knowledge of verb alternations (change-of-state and object-drop constructions in English, German and Italian, and Hebrew binyanim). The datasets comprise thousands of the Blackbird Language Matrices (BLMs) problems. The BLM task -- an RPM/ARC-like task devised specifically for language -- is a controlled linguistic puzzle where models must select the sentence that completes a pattern according to syntactic and semantic rules. We introduce three types of templates varying in complexity and apply linguistically-informed data augmentation strategies across synthetic and natural data. We provide simple baseline performance results across English, Italian, German, and Hebrew, that demonstrate the diagnostic usefulness of the datasets.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 17 of its 56 references that are in the map. First posted on arXiv 16 Mar 2026.
Builds on
17 of 56 references.
References · 17
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
- 2020Syntactic Structure from Deep Learningcited by 99
- 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
- 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
- 2018Verb Argument Structure Alternations in Word and Sentence Embeddingscited by 25
- 2023Bayes in the age of intelligent machinescited by 22
- 2022A Second Wave of UD Hebrew Treebanking and Cross-Domain Parsingcited by 10
- 2023Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specificationscited by 9
- 2024Tracking linguistic information in transformer-based sentence embeddings through targeted sparsificationcited by 6
5 more
- 2024Exploring syntactic information in sentence embeddings through multilingual subject-verb agreementcited by 6
- 2024Exploring Italian sentence embeddings properties through multi-taskingcited by 6
- 2025Does Training on Synthetic Data Make Models Less Robust?cited by 3
- 2026Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrewcited by 2
- 2025Analogical Structure, Minimal Contextual Cues and Contrastive Distractors: Input Design for Sample-Efficient Linguistic Rule Inductioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.