# Something Old, Something New: Grammar-based CCG Parsing with Transformer Models

arXiv: https://arxiv.org/abs/2109.10044 · HTML: https://mlsanity.com/paper/2109.10044/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 81 references that are in the map.
- First posted on arXiv 21 Sep 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://mlsanity.com/paper/1809.08370/) (2018)
- [QNLP in Practice: Running Compositional Models of Meaning on a Quantum Computer](https://mlsanity.com/paper/2102.12846/) (2021)
- [A Minimal Span-Based Neural Constituency Parser](https://mlsanity.com/paper/1705.03919/) (2017)
- [A* CCG Parsing with a Supertag and Dependency Factored Model](https://mlsanity.com/paper/1704.06936/) (2017)
- [A CCG-Based Version of the DisCoCat Framework](https://mlsanity.com/paper/2105.07720/) (2021)
- [Supertagging Combinatory Categorial Grammar with Attentive Graph Convolutional Networks](https://mlsanity.com/paper/2010.06115/) (2020)
- [Global Neural CCG Parsing with Optimality Guarantees](https://mlsanity.com/paper/1607.01432/) (2016)
- [End-to-end Graph-based TAG Parsing with Neural Networks](https://mlsanity.com/paper/1804.06610/) (2018)
- [On the Complexity of CCG Parsing](https://mlsanity.com/paper/1702.06594/) (2017)

## Built on by (most cited first, 6 of 6)

- [lambeq: An Efficient High-Level Python Library for Quantum NLP](https://mlsanity.com/paper/2110.04236/) (2021)
- [A Pipeline For Discourse Circuits From CCG](https://mlsanity.com/paper/2311.17892/) (2023)
- [DisCoCLIP: A Distributional Compositional Tensor Network Encoder for Vision-Language Understanding](https://mlsanity.com/paper/2509.21287/) (2025)
- [Structural Ambiguity and its Disambiguation in Language Model Based Parsers: the Case of Dutch Clause Relativization](https://mlsanity.com/paper/2305.14917/) (2023)
- [SPINDLE: Spinning Raw Text into Lambda Terms with Graph Attention](https://mlsanity.com/paper/2302.12050/) (2023)
- [Towards Transparency in Coreference Resolution: A Quantum-Inspired Approach](https://mlsanity.com/paper/2312.00688/) (2023)

## Abstract (arXiv)

This report describes the parsing problem for Combinatory Categorial Grammar (CCG), showing how a combination of Transformer-based neural models and a symbolic CCG grammar can lead to substantial gains over existing approaches. The report also documents a 20-year research program, showing how NLP methods have evolved over this time. The staggering accuracy improvements provided by neural models for CCG parsing can be seen as a reflection of the improvements seen in NLP more generally. The report provides a minimal introduction to CCG and CCG parsing, with many pointers to the relevant literature. It then describes the CCG supertagging problem, and some recent work from Tian et al. (2020) which applies Transformer-based models to supertagging with great effect. I use this existing model to develop a CCG multitagger, which can serve as a front-end to an existing CCG parser. Simply using this new multitagger provides substantial gains in parsing accuracy. I then show how a Transformer-based model from the parsing literature can be combined with the grammar-based CCG parser, setting a new state-of-the-art for the CCGbank parsing task of almost 93% F-score for labelled dependencies, with complete sentence accuracies of over 50%.