# Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Motivations and Formal Specifications

arXiv: https://arxiv.org/abs/2306.11444 · HTML: https://mlsanity.com/paper/2306.11444/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 26 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 8 of 8)

- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks](https://mlsanity.com/paper/2205.10866/) (2022)

## Built on by (most cited first, 9 of 9)

- [Are there identifiable structural parts in the sentence embedding whole?](https://mlsanity.com/paper/2406.16563/) (2024)
- [Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification](https://mlsanity.com/paper/2407.18119/) (2024)
- [Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement](https://mlsanity.com/paper/2409.06567/) (2024)
- [Exploring Italian sentence embeddings properties through multi-tasking](https://mlsanity.com/paper/2409.06622/) (2024)
- [Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://mlsanity.com/paper/2602.05648/) (2026)
- [Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies](https://mlsanity.com/paper/2603.15295/) (2026)
- [Disentangling continuous and discrete linguistic signals in transformer-based sentence embeddings](https://mlsanity.com/paper/2312.11272/) (2023)
- [Blackbird Language Matrices: A Framework to Investigate the Linguistic Competence of Language Models](https://mlsanity.com/paper/2602.20966/) (2026)
- [Generalization through Lexical Abstraction in Transformer Models: The Case of Functional Words](https://mlsanity.com/paper/2609.19887/) (2026)

## Abstract (arXiv)

We motivate and formally define a new task for fine-tuning rule-like generalization in large language models. It is conjectured that the shortcomings of current LLMs are due to a lack of ability to generalize. It has been argued that, instead, humans are better at generalization because they have a tendency at extracting rules from complex data. We try to recreate this tendency to rule-based generalization. When exposed to tests of analytic intelligence, for example, the visual RAVEN IQ test, human problem-solvers identify the relevant objects in the picture and their relevant attributes and reason based on rules applied to these objects and attributes. Based on the induced rules, they are able to provide a solution to the test. We propose a task that translates this IQ task into language. In this paper, we provide the formal specification for the task and the generative process of its datasets.