# Contextual trajectory and incremental contextual displacement: Towards using LLMs to understand dynamic, utterance-specific meaning construction

arXiv: https://arxiv.org/abs/2610.00840 · HTML: https://mlsanity.com/paper/2610.00840/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 61 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve](https://mlsanity.com/paper/2309.13638/) (2023)
- [Does Prompt Formatting Have Any Impact on LLM Performance?](https://mlsanity.com/paper/2411.10541/) (2024)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [What is a Number, That a Large Language Model May Know It?](https://mlsanity.com/paper/2502.01540/) (2025)
- [Incremental Comprehension of Garden-Path Sentences by Large Language Models: Semantic Interpretation, Syntactic Re-Analysis, and Attention](https://mlsanity.com/paper/2405.16042/) (2024)
- [Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning](https://mlsanity.com/paper/2412.10924/) (2024)
- [Lexinvariant Language Models](https://mlsanity.com/paper/2305.16349/) (2023)
- [Quantifying Lexical Semantic Shift via Unbalanced Optimal Transport](https://mlsanity.com/paper/2412.12569/) (2024)
- [Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space](https://mlsanity.com/paper/2605.12412/) (2026)
- [Practical and Ready-to-Use Methodology to Assess the re-identification Risk in Anonymized Datasets](https://mlsanity.com/paper/2501.10841/) (2025)
- [The Last Fingerprint: How Markdown Training Shapes LLM Prose](https://mlsanity.com/paper/2603.27006/) (2026)
- [Improving the Coverage and the Generalization Ability of Neural Word Sense Disambiguation through Hypernymy and Hyponymy Relationships](https://mlsanity.com/paper/1811.00960/) (2018)
- [Revisiting Real-Time Digging-In Effects: No Evidence from NP/Z Garden-Paths](https://mlsanity.com/paper/2603.23624/) (2026)
- [LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks](https://mlsanity.com/paper/2609.34187/) (2026)

## Abstract (arXiv)

Transformer-based large language models (LLMs) such as RoBERTa represent text using contextual word embeddings (CWEs), which alter the embeddings associated with each token based on surrounding context. We construct token-wise incremental trajectories by repeatedly recomputing a token's CWE as successive words are added to a sentence, yielding a representation of how contextualized embeddings evolve as the utterance unfolds. We evaluate this approach using garden-path sentences as a test case with characteristic features. Token-wise trajectories reproduce known features of garden-path processing, including disruption around the critical region, and reliably distinguish garden-path sentences from matched disambiguated controls. We introduce several metrics for quantifying representational displacement across contextual increments and show that trajectory information can be highly predictive of sentence type. We find that ambiguity-related information is recoverable not only from the sentence-level CLS representation but also from ordinary vocabulary tokens, suggesting that utterance-level information is distributed across multiple representational scales. In exploratory analyses, we find qualitatively similar trajectory structures in other ambiguity- and misdirection-related linguistic phenomena. Together, these results establish token-wise incremental trajectories as a promising framework for studying utterance-specific meaning construction using LLMs.