# Schr"odinger's Tree -- On Syntax and Neural Language Models arXiv: https://arxiv.org/abs/2110.08887 ยท HTML: https://mlsanity.com/paper/2110.08887/ ## Key facts - Cited by 6 papers in the map (data through 14 Oct 2026). - Builds on 67 of its 115 references that are in the map. - First posted on arXiv 17 Oct 2021. ## Builds on (most cited first, 30 of 67) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017) - ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016) - [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018) - [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013) - [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017) - [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019) - [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017) - [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019) - [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019) - [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019) - [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019) - [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021) - [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020) - [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018) - [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019) - [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019) - [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019) - [e-SNLI: Natural Language Inference with Natural Language Explanations](https://mlsanity.com/paper/1812.01193/) (2018) - [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019) - [Is Attention Interpretable?](https://mlsanity.com/paper/1906.03731/) (2019) - [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019) - [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019) - [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016) - [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018) - [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019) - [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020) ## Built on by (most cited first, 6 of 6) - [Word Order Does Matter (And Shuffled Language Models Know It)](https://mlsanity.com/paper/2203.10995/) (2022) - [Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models](https://mlsanity.com/paper/2305.14775/) (2023) - [Assessment of Pre-Trained Models Across Languages and Grammars](https://mlsanity.com/paper/2309.11165/) (2023) - [Sampling Data with Chains of Forward-Backward Diffusion Steps](https://mlsanity.com/paper/2605.27006/) (2026) - [Local Structure Matters Most in Most Languages](https://mlsanity.com/paper/2211.05025/) (2022) - [Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations](https://mlsanity.com/paper/2401.14212/) (2024) ## Abstract (arXiv) In the last half-decade, the field of natural language processing (NLP) has undergone two major transitions: the switch to neural networks as the primary modeling paradigm and the homogenization of the training regime (pre-train, then fine-tune). Amidst this process, language models have emerged as NLP's workhorse, displaying increasingly fluent generation capabilities and proving to be an indispensable means of knowledge transfer downstream. Due to the otherwise opaque, black-box nature of such models, researchers have employed aspects of linguistic theory in order to characterize their behavior. Questions central to syntax -- the study of the hierarchical structure of language -- have factored heavily into such work, shedding invaluable insights about models' inherent biases and their ability to make human-like generalizations. In this paper, we attempt to take stock of this growing body of literature. In doing so, we observe a lack of clarity across numerous dimensions, which influences the hypotheses that researchers form, as well as the conclusions they draw from their findings. To remedy this, we urge researchers make careful considerations when investigating coding properties, selecting representations, and evaluating via downstream tasks. Furthermore, we outline the implications of the different types of research questions exhibited in studies on syntax, as well as the inherent pitfalls of aggregate metrics. Ultimately, we hope that our discussion adds nuance to the prospect of studying language models and paves the way for a less monolithic perspective on syntax in this context.