# Directional Contextual Representations for Dependency Relations: Why Cross-Direction Pairing Fails

arXiv: https://arxiv.org/abs/2608.20647 · HTML: https://mlsanity.com/paper/2608.20647/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 4 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 4 of 4)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)

## Abstract (arXiv)

Splitting a bidirectional LSTM's contextual representation into a forward-only Fi (strictly a function of tokens 1..i) and a backward-only Bi (strictly a function of tokens i..n) beats either alone and beats a fused self-attention representation for dependency relation-type classification. But a specific, natural extension of this idea -- pairing a token's forward state against a candidate's backward state (``cross-direction'' pairing, Fi vs. Bj) -- consistently underperforms same-direction pairing, and the penalty grows, not shrinks, with token distance, both paired-bootstrap significant. We diagnose why using a frozen-trunk methodology: architectural information leakage between directions is impossible by construction (a single-layer BiLSTM, verified by code inspection); 93% of the same-vs-cross gap survives freezing the trunk and training only fresh heads, ruling out training-co-adaptation as the primary cause; linear regression shows partial representational redundancy between Fi and Bi (R2=0.324 vs. 0.028 for a shuffled control) and a linear probe shows partial anticipatory encoding of upcoming tokens in Fi (36.5% vs. 17.2% majority baseline) -- real effects, but neither alone, nor combined, cleanly explains the full gap. Extended frozen-trunk diagnostics (a positional probe and a distance-decay probe) show directional information is genuinely stored but not exactly positioned, and propagates only a few tokens before decaying to baseline -- consistent with, and mechanistically underneath, the distance-growth finding.