# Probing for Incremental Parse States in Autoregressive Language Models

arXiv: https://arxiv.org/abs/2211.09748 · HTML: https://mlsanity.com/paper/2211.09748/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://mlsanity.com/paper/1905.06316/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior](https://mlsanity.com/paper/2006.01912/) (2020)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021)
- [Finding Syntax in Human Encephalography with Beam Search](https://mlsanity.com/paper/1806.04127/) (2018)
- [What if This Modified That? Syntactic Interventions via Counterfactual Embeddings](https://mlsanity.com/paper/2105.14002/) (2021)
- [Can neural networks acquire a structural bias from raw linguistic data?](https://mlsanity.com/paper/2007.06761/) (2020)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans](https://mlsanity.com/paper/2106.03232/) (2021)
- [Effective Inference for Generative Neural Parsing](https://mlsanity.com/paper/1707.08976/) (2017)
- [Structural Guidance for Transformer Language Models](https://mlsanity.com/paper/2108.00104/) (2021)
- [Structural Supervision Improves Learning of Non-Local Grammatical Dependencies](https://mlsanity.com/paper/1903.00943/) (2019)
- [The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generation](https://mlsanity.com/paper/2109.07848/) (2021)
- [Effective Batching for Recurrent Neural Network Grammars](https://mlsanity.com/paper/2105.14822/) (2021)
- [When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probes](https://mlsanity.com/paper/2204.09722/) (2022)
- [Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans](https://mlsanity.com/paper/2110.07240/) (2021)
- [Probing for Labeled Dependency Trees](https://mlsanity.com/paper/2203.12971/) (2022)
- [Arc-swift: A Novel Transition System for Dependency Parsing](https://mlsanity.com/paper/1705.04434/) (2017)

## Built on by (most cited first, 24 of 24)

- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [We Can't Understand AI Using our Existing Vocabulary](https://mlsanity.com/paper/2502.07586/) (2025)
- [Backpack Language Models](https://mlsanity.com/paper/2305.16765/) (2023)
- [SyntaxShap: Syntax-aware Explainability Method for Text Generation](https://mlsanity.com/paper/2402.09259/) (2024)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://mlsanity.com/paper/2506.16678/) (2025)
- [Are Structural Concepts Universal in Transformer Language Models? Towards Interpretable Cross-Lingual Generalization](https://mlsanity.com/paper/2310.12794/) (2023)
- [Multilingual Nonce Dependency Treebanks: Understanding how Language Models represent and process syntactic structure](https://mlsanity.com/paper/2311.07497/) (2023)
- [Probing Syntax in Large Language Models: Successes and Remaining Challenges](https://mlsanity.com/paper/2508.03211/) (2025)
- [Incremental Sentence Processing Mechanisms in Autoregressive Transformer Language Models](https://mlsanity.com/paper/2412.05353/) (2024)
- [Approximating CKY with Transformers](https://mlsanity.com/paper/2305.02386/) (2023)
- [Trick or Neat: Adversarial Ambiguity and Language Model Evaluation](https://mlsanity.com/paper/2506.01205/) (2025)
- [On the Challenges of Fully Incremental Neural Dependency Parsing](https://mlsanity.com/paper/2309.16254/) (2023)
- [Polar probe linearly decodes semantic structures from LLMs](https://mlsanity.com/paper/2605.14125/) (2026)
- [When Only Time Will Tell: Interpreting How Transformers Process Local Ambiguities Through the Lens of Restart-Incrementality](https://mlsanity.com/paper/2402.13113/) (2024)
- [The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models](https://mlsanity.com/paper/2601.19926/) (2026)
- [Implicit Representations of Grammaticality in Language Models](https://mlsanity.com/paper/2605.05197/) (2026)
- [Linear representations of grammaticality in neural language models](https://mlsanity.com/paper/2607.15175/) (2026)
- [Multipath parsing in the brain](https://mlsanity.com/paper/2401.18046/) (2024)
- [Derivational Probing: Unveiling the Layer-wise Derivation of Syntactic Structures in Neural Language Models](https://mlsanity.com/paper/2506.21861/) (2025)
- [Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs](https://mlsanity.com/paper/2602.15846/) (2026)
- [A Mechanism and Optimization Study on the Impact of Information Density on User-Generated Content Named Entity Recognition](https://mlsanity.com/paper/2604.18944/) (2026)
- [Why are language models less surprised than humans? Testing the Parse Multiplicity Mismatch Hypothesis](https://mlsanity.com/paper/2605.15440/) (2026)
- [Representation of syntax in LLMs through the lens of linear distance and similarity-aware entropy](https://mlsanity.com/paper/2608.27813/) (2026)

## Abstract (arXiv)

Next-word predictions from autoregressive neural language models show remarkable sensitivity to syntax. This work evaluates the extent to which this behavior arises as a result of a learned ability to maintain implicit representations of incremental syntactic structures. We extend work in syntactic probing to the incremental setting and present several probes for extracting incomplete syntactic structure (operationalized through parse states from a stack-based parser) from autoregressive language models. We find that our probes can be used to predict model preferences on ambiguous sentence prefixes and causally intervene on model representations and steer model behavior. This suggests implicit incremental syntactic inferences underlie next-word predictions in autoregressive neural language models.