# Improving Constituency Parsing with Span Attention

arXiv: https://arxiv.org/abs/2010.07543 · HTML: https://mlsanity.com/paper/2010.07543/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders](https://mlsanity.com/paper/2105.01279/) (2021)
- [Bottom-Up Constituency Parsing and Nested Named Entity Recognition with Pointer Networks](https://mlsanity.com/paper/2110.05419/) (2021)
- [BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsing](https://mlsanity.com/paper/2206.10668/) (2022)
- [Improving Image Captioning via Predicting Structured Concepts](https://mlsanity.com/paper/2311.08223/) (2023)
- [Investigating Non-local Features for Neural Constituency Parsing](https://mlsanity.com/paper/2109.12814/) (2021)
- [Discontinuous Grammar as a Foreign Language](https://mlsanity.com/paper/2110.10431/) (2021)
- [Approximating CKY with Transformers](https://mlsanity.com/paper/2305.02386/) (2023)
- [Head-driven Phrase Structure Parsing in O(n3) Time Complexity](https://mlsanity.com/paper/2105.09835/) (2021)
- [Constituency Parsing using LLMs](https://mlsanity.com/paper/2310.19462/) (2023)
- [To be Continuous, or to be Discrete, Those are Bits of Questions](https://mlsanity.com/paper/2406.07812/) (2024)
- [Fast Rule-Based Decoding: Revisiting Syntactic Rules in Neural Constituency Parsing](https://mlsanity.com/paper/2212.08458/) (2022)
- [Sentiment Analysis Dataset in Moroccan Dialect: Bridging the Gap Between Arabic and Latin Scripted dialect](https://mlsanity.com/paper/2303.15987/) (2023)
- [Nested Named Entity Recognition as Holistic Structure Parsing](https://mlsanity.com/paper/2204.08006/) (2022)
- [Entity-Aware Biaffine Attention Model for Improved Constituent Parsing with Reduced Entity Violations](https://mlsanity.com/paper/2409.00625/) (2024)
- [Scaling Graph-Based Dependency Parsing with Arc Vectorization and Attention-Based Refinement](https://mlsanity.com/paper/2501.09451/) (2025)
- [Contrastive Learning on LLM Back Generation Treebank for Cross-domain Constituency Parsing](https://mlsanity.com/paper/2505.20976/) (2025)
- [Exploiting Pre-trained Encoder-Decoder Transformers for Sequence-to-Sequence Constituent Parsing](https://mlsanity.com/paper/2605.13373/) (2026)

## Abstract (arXiv)

Constituency parsing is a fundamental and important task for natural language understanding, where a good representation of contextual information can help this task. N-grams, which is a conventional type of feature for contextual information, have been demonstrated to be useful in many tasks, and thus could also be beneficial for constituency parsing if they are appropriately modeled. In this paper, we propose span attention for neural chart-based constituency parsing to leverage n-gram information. Considering that current chart-based parsers with Transformer-based encoder represent spans by subtraction of the hidden states at the span boundaries, which may cause information loss especially for long spans, we incorporate n-grams into span representations by weighting them according to their contributions to the parsing process. Moreover, we propose categorical span attention to further enhance the model by weighting n-grams within different length categories, and thus benefit long-sentence parsing. Experimental results on three widely used benchmark datasets demonstrate the effectiveness of our approach in parsing Arabic, Chinese, and English, where state-of-the-art performance is obtained by our approach on all of them.