# Supertagging the Long Tail with Tree-Structured Decoding of Complex Categories

arXiv: https://arxiv.org/abs/2012.01285 · HTML: https://mlsanity.com/paper/2012.01285/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2020.

## Built on by (most cited first, 11 of 11)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [Language Models Learn Rare Phenomena from Less Rare Phenomena: The Case of the Missing AANNs](https://mlsanity.com/paper/2403.19827/) (2024)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Improving BERT Pretraining with Syntactic Supervision](https://mlsanity.com/paper/2104.10516/) (2021)
- [Revisiting Supertagging for Faster HPSG Pasing](https://mlsanity.com/paper/2309.07590/) (2023)
- [Geometry-Aware Supertagging with Heterogeneous Dynamic Convolutions](https://mlsanity.com/paper/2203.12235/) (2022)
- [Generating CCG Categories](https://mlsanity.com/paper/2103.08139/) (2021)
- [Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets](https://mlsanity.com/paper/2109.07494/) (2021)
- [SPINDLE: Spinning Raw Text into Lambda Terms with Graph Attention](https://mlsanity.com/paper/2302.12050/) (2023)
- [Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures](https://mlsanity.com/paper/2305.18915/) (2023)

## Abstract (arXiv)

Although current CCG supertaggers achieve high accuracy on the standard WSJ test set, few systems make use of the categories' internal structure that will drive the syntactic derivation during parsing. The tagset is traditionally truncated, discarding the many rare and complex category types in the long tail. However, supertags are themselves trees. Rather than give up on rare tags, we investigate constructive models that account for their internal structure, including novel methods for tree-structured prediction. Our best tagger is capable of recovering a sizeable fraction of the long-tail supertags and even generates CCG categories that have never been seen in training, while approximating the prior state of the art in overall tag accuracy with fewer parameters. We further investigate how well different approaches generalize to out-of-domain evaluation sets.