# Discontinuous Constituency and BERT: A Case Study of Dutch

arXiv: https://arxiv.org/abs/2203.01063 · HTML: https://mlsanity.com/paper/2203.01063/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 37 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Open Sesame: Getting Inside BERT's Linguistic Knowledge](https://mlsanity.com/paper/1906.01698/) (2019)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [SICKNL: A Dataset for Dutch Natural Language Inference](https://mlsanity.com/paper/2101.05716/) (2021)
- [Parsing as Pretraining](https://mlsanity.com/paper/2002.01685/) (2020)
- [The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews](https://mlsanity.com/paper/1910.00896/) (2019)

## Built on by (most cited first, 3 of 3)

- [Word-order typology in Multilingual BERT: A case study in subordinate-clause detection](https://mlsanity.com/paper/2205.11987/) (2022)
- [Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativization](https://mlsanity.com/paper/2305.14917/) (2023)
- [Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages](https://mlsanity.com/paper/2309.00857/) (2023)

## Abstract (arXiv)

In this paper, we set out to quantify the syntactic capacity of BERT in the evaluation regime of non-context free patterns, as occurring in Dutch. We devise a test suite based on a mildly context-sensitive formalism, from which we derive grammars that capture the linguistic phenomena of control verb nesting and verb raising. The grammars, paired with a small lexicon, provide us with a large collection of naturalistic utterances, annotated with verb-subject pairings, that serve as the evaluation test bed for an attention-based span selection probe. Our results, backed by extensive analysis, suggest that the models investigated fail in the implicit acquisition of the dependencies examined.