# GROOT: Corrective Reward Optimization for Generative Sequential Labeling

arXiv: https://arxiv.org/abs/2209.14694 · HTML: https://mlsanity.com/paper/2209.14694/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 29 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Sequence Level Training with Recurrent Neural Networks](https://mlsanity.com/paper/1511.06732/) (2015)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Minimum Risk Training for Neural Machine Translation](https://mlsanity.com/paper/1512.02433/) (2015)
- [On the Weaknesses of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1907.01752/) (2019)
- [Grammar as a Foreign Language](https://mlsanity.com/paper/1412.7449/) (2014)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [Augmented Natural Language for Generative Sequence Labeling](https://mlsanity.com/paper/2009.13272/) (2020)
- [Transforming Sequence Tagging Into A Seq2Seq Task](https://mlsanity.com/paper/2203.08378/) (2022)
- [The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family](https://mlsanity.com/paper/1604.08859/) (2016)
- [Reward Optimization for Neural Machine Translation with Learned Metrics](https://mlsanity.com/paper/2104.07541/) (2021)
- [STIL -- Simultaneous Slot Filling, Translation, Intent Classification, and Language Identification: Initial Results using mBART on MultiATIS++](https://mlsanity.com/paper/2010.00760/) (2020)

## Built on by (most cited first, 1 of 1)

- [How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?](https://mlsanity.com/paper/2212.10767/) (2022)

## Abstract (arXiv)

Sequential labeling is a fundamental NLP task, forming the backbone of many applications. Supervised learning of Seq2Seq models has shown great success on these problems. However, the training objectives are still significantly disconnected with the metrics and desiderata we care about in practice. For example, a practical sequence tagging application may want to optimize for a certain precision-recall trade-off (of the top-k predictions) which is quite different from the standard objective of maximizing the likelihood of the gold labeled sequence. Thus to bridge this gap, we propose GROOT -- a simple yet effective framework for Generative Reward Optimization Of Text sequences. GROOT works by training a generative sequential labeling model to match the decoder output distribution with that of the (black-box) reward function. Using an iterative training regime, we first generate prediction candidates, then correct errors in them, and finally contrast those candidates (based on their reward values). As demonstrated via extensive experiments on four public benchmarks, GROOT significantly improves all reward metrics. Furthermore, GROOT leads to improvements of the overall decoder distribution as evidenced by the quality gains of the top-k candidates.