# Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsing

arXiv: https://arxiv.org/abs/2605.13373 · HTML: https://mlsanity.com/paper/2605.13373/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 57 references that are in the map.
- First posted on arXiv 13 May 2026.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [Recurrent Neural Network Grammars](https://mlsanity.com/paper/1602.07776/) (2016)
- [Grammar as a Foreign Language](https://mlsanity.com/paper/1412.7449/) (2014)
- [Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees](https://mlsanity.com/paper/2103.04350/) (2021)
- [GottBERT: a pure German Language Model](https://mlsanity.com/paper/2012.02110/) (2020)
- [Transformers with convolutional context for ASR](https://mlsanity.com/paper/1904.11660/) (2019)
- [Head-Driven Phrase Structure Grammar Parsing on Penn Treebank](https://mlsanity.com/paper/1907.02684/) (2019)
- [Transition-based Parsing with Stack-Transformers](https://mlsanity.com/paper/2010.10669/) (2020)
- [SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics](https://mlsanity.com/paper/2005.04114/) (2020)
- [Syntactic Scaffolds for Semantic Structures](https://mlsanity.com/paper/1808.10485/) (2018)
- [In-Order Transition-based Constituent Parsing](https://mlsanity.com/paper/1707.05000/) (2017)
- [Strongly Incremental Constituency Parsing with Graph Neural Networks](https://mlsanity.com/paper/2010.14568/) (2020)
- [Span-Based Constituency Parsing with a Structure-Label System and Provably Optimal Dynamic Oracles](https://mlsanity.com/paper/1612.06475/) (2016)
- [A Tree-based Decoder for Neural Machine Translation](https://mlsanity.com/paper/1808.09374/) (2018)
- [Improving Constituency Parsing with Span Attention](https://mlsanity.com/paper/2010.07543/) (2020)
- [Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation](https://mlsanity.com/paper/2006.05474/) (2020)
- [Multitask Pointer Network for Multi-Representational Parsing](https://mlsanity.com/paper/2009.09730/) (2020)
- [How to best use Syntax in Semantic Role Labelling](https://mlsanity.com/paper/1906.00266/) (2019)
- [Span-based discontinuous constituency parsing: a family of exact chart-based algorithms with time complexities from O(n^6) down to O(n^3)](https://mlsanity.com/paper/2003.13785/) (2020)
- [Enriched In-Order Linearization for Faster Sequence-to-Sequence Constituent Parsing](https://mlsanity.com/paper/2005.13334/) (2020)
- [Faster Shift-Reduce Constituent Parsing with a Non-Binary, Bottom-Up Strategy](https://mlsanity.com/paper/1804.07961/) (2018)
- [Encoder-Decoder Shift-Reduce Syntactic Parsing](https://mlsanity.com/paper/1706.07905/) (2017)
- [Discontinuous Constituent Parsing with Pointer Networks](https://mlsanity.com/paper/2002.01824/) (2020)

## Abstract (arXiv)

To achieve deep natural language understanding, syntactic constituent parsing plays a crucial role and is widely required by many artificial intelligence systems for processing both text and speech. A recent approach involves using standard sequence-to-sequence models to handle constituent parsing as a machine translation problem, moving away from traditional task-specific parsers. These models are typically initialized with pre-trained encoder-only language models like BERT or RoBERTa. However, the use of pre-trained encoder-decoder language models for constituency parsing has not been thoroughly explored. To bridge this gap, we extend the sequence-to-sequence framework by investigating parsers built on pre-trained encoder-decoder architectures, including BART, mBART, and T5. We fine-tune them to generate linearized parse trees and extensively evaluate them on different linearization strategies across both continuous treebanks and more complex discontinuous benchmarks. Our results demonstrate that our approach outperforms all prior sequence-to-sequence models and performs competitively with leading task-specific constituent parsers on continuous constituent parsing.