# Transforming Sequence Tagging Into A Seq2Seq Task

arXiv: https://arxiv.org/abs/2203.08378 · HTML: https://mlsanity.com/paper/2203.08378/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Structured Prediction as Translation between Augmented Natural Languages](https://mlsanity.com/paper/2101.05779/) (2021)
- [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://mlsanity.com/paper/2008.09335/) (2020)
- [A Unified Generative Framework for Various NER Subtasks](https://mlsanity.com/paper/2106.01223/) (2021)
- [Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections](https://mlsanity.com/paper/2104.04670/) (2021)
- [Cross-Lingual Transfer Learning for Multilingual Task Oriented Dialog](https://mlsanity.com/paper/1810.13327/) (2018)
- [LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5](https://mlsanity.com/paper/2110.07298/) (2021)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [Augmented Natural Language for Generative Sequence Labeling](https://mlsanity.com/paper/2009.13272/) (2020)
- [Multilingual Code-Switching for Zero-Shot Cross-Lingual Intent Prediction and Slot Filling](https://mlsanity.com/paper/2103.07792/) (2021)
- [Intent Classification and Slot Filling for Privacy Policies](https://mlsanity.com/paper/2101.00123/) (2021)
- [A Sequence-to-Sequence Model for Semantic Role Labeling](https://mlsanity.com/paper/1807.03006/) (2018)

## Built on by (most cited first, 15 of 15)

- [Why Does Zero-Shot Cross-Lingual Generation Fail? An Explanation and a Solution](https://mlsanity.com/paper/2305.17325/) (2023)
- [CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsing](https://mlsanity.com/paper/2210.07074/) (2022)
- [LLMs as Repositories of Factual Knowledge: Limitations and Solutions](https://mlsanity.com/paper/2501.12774/) (2025)
- [HyperLoader: Integrating Hypernetwork-Based LoRA and Adapter Layers into Multi-Task Transformers for Sequence Labelling](https://mlsanity.com/paper/2407.01411/) (2024)
- [Deconstructing In-Context Learning: Understanding Prompts via Corruption](https://mlsanity.com/paper/2404.02054/) (2024)
- [Distillation of encoder-decoder transformers for sequence labelling](https://mlsanity.com/paper/2302.05454/) (2023)
- [Chain of Thought Explanation for Dialogue State Tracking](https://mlsanity.com/paper/2403.04656/) (2024)
- [Style Transfer as Data Augmentation: A Case Study on Named Entity Recognition](https://mlsanity.com/paper/2210.07916/) (2022)
- [Optimal Transport Posterior Alignment for Cross-lingual Semantic Parsing](https://mlsanity.com/paper/2307.04096/) (2023)
- [Take One Step at a Time to Know Incremental Utility of Demonstration: An Analysis on Reranking for Few-Shot In-Context Learning](https://mlsanity.com/paper/2311.09619/) (2023)
- [GROOT: Corrective Reward Optimization for Generative Sequential Labeling](https://mlsanity.com/paper/2209.14694/) (2022)
- [How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?](https://mlsanity.com/paper/2212.10767/) (2022)
- [ImPaKT: A Dataset for Open-Schema Knowledge Base Construction](https://mlsanity.com/paper/2212.10770/) (2022)
- [Deepparse : An Extendable, and Fine-Tunable State-Of-The-Art Library for Parsing Multinational Street Addresses](https://mlsanity.com/paper/2311.11846/) (2023)
- [The Multilingual FrameNet Corpus](https://mlsanity.com/paper/2608.23037/) (2026)

## Abstract (arXiv)

Pretrained, large, generative language models (LMs) have had great success in a wide range of sequence tagging and structured prediction tasks. Casting a sequence tagging task as a Seq2Seq one requires deciding the formats of the input and output sequences. However, we lack a principled understanding of the trade-offs associated with these formats (such as the effect on model accuracy, sequence length, multilingual generalization, hallucination). In this paper, we rigorously study different formats one could use for casting input text sentences and their output labels into the input and target (i.e., output) of a Seq2Seq model. Along the way, we introduce a new format, which we show to to be both simpler and more effective. Additionally the new format demonstrates significant gains in the multilingual settings -- both zero-shot transfer learning and joint training. Lastly, we find that the new format is more robust and almost completely devoid of hallucination -- an issue we find common in existing formats. With well over a 1000 experiments studying 14 different formats, over 7 diverse public benchmarks -- including 3 multilingual datasets spanning 7 languages -- we believe our findings provide a strong empirical basis in understanding how we should tackle sequence tagging tasks.