# ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks

arXiv: https://arxiv.org/abs/2401.16589 · HTML: https://mlsanity.com/paper/2401.16589/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 29 Jan 2024.

## Builds on (most cited first, 30 of 35)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [Template-Based Named Entity Recognition Using BART](https://mlsanity.com/paper/2106.01760/) (2021)
- [The Multilingual Amazon Reviews Corpus](https://mlsanity.com/paper/2010.02573/) (2020)
- [mGPT: Few-Shot Learners Go Multilingual](https://mlsanity.com/paper/2204.07580/) (2022)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)
- [Template-free Prompt Tuning for Few-shot NER](https://mlsanity.com/paper/2109.13532/) (2021)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [Discrete and Soft Prompting for Multilingual Models](https://mlsanity.com/paper/2109.03630/) (2021)
- [Revisiting the Primacy of English in Zero-shot Cross-lingual Transfer](https://mlsanity.com/paper/2106.16171/) (2021)
- [Prompting Language Models for Linguistic Structure](https://mlsanity.com/paper/2211.07830/) (2022)
- [Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt](https://mlsanity.com/paper/2202.11451/) (2022)

## Built on by (most cited first, 7 of 7)

- [BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning](https://mlsanity.com/paper/2406.17764/) (2024)
- [Evaluating Zero-Shot Multilingual Aspect-Based Sentiment Analysis with Large Language Models](https://mlsanity.com/paper/2412.12564/) (2024)
- [Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers](https://mlsanity.com/paper/2402.11700/) (2024)
- [Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models](https://mlsanity.com/paper/2402.18397/) (2024)
- [GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network](https://mlsanity.com/paper/2402.11709/) (2024)
- [Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models](https://mlsanity.com/paper/2506.01592/) (2025)
- [Unveiling Factors for Enhanced POS Tagging: A Study of Low-Resource Medieval Romance Languages](https://mlsanity.com/paper/2506.17715/) (2025)

## Abstract (arXiv)

Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.