mlsanityData to 14 Oct 2026

Does Traversal Order Matter? A Systematic Study of Tree Traversal Methods in Transformer Grammars

arXiv 2606.16836 ↗First posted 15 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformer Grammars (TGs) enhance language modeling by incorporating syntactic tree structures. Despite the potentially significant impact on model performance of how syntactic trees are linearized in TGs, existing studies rely solely on Depth-First Traversal (DFT) for linearization. In this paper, we expand the traversal design space by exploring Breadth-First Traversal (BFT) and a novel hybrid traversal strategy, Production-Rule Traversal (PRT), which combines the structural lookahead of BFT with the early lexical generation of DFT. We integrate these traversal methods with varying tree configurations and masking strategies, and empirically evaluate their performance on language modeling, syntactic generalization and summarization. We reveal the inherent trade-offs between nested composition and global lookahead, providing actionable recommendations for designing task-aware Transformer Grammars.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 19 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

15 of 19 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2020GLU Variants Improve Transformercited by 1,804
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  8. 2016Recurrent Neural Network Grammarscited by 109
  9. 2020Fast and Accurate Neural CRF Constituency Parsingcited by 50
  10. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 41
  11. 2017Effective Inference for Generative Neural Parsingcited by 25
  12. 2021Structural Guidance for Transformer Language Modelscited by 25
3 more
  1. 2017In-Order Transition-based Constituent Parsingcited by 23
  2. 2024Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scalecited by 10
  3. 2024Dependency Transformer Grammars: Integrating Dependency Structures into Transformer Language Modelscited by 8

Built on by

Not cited yet.