# Bringing Emerging Architectures to Sequence Labeling in NLP

arXiv: https://arxiv.org/abs/2509.25918 · HTML: https://mlsanity.com/paper/2509.25918/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 75 references that are in the map.
- First posted on arXiv 30 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Deep Biaffine Attention for Neural Dependency Parsing](https://mlsanity.com/paper/1611.01734/) (2016)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control](https://mlsanity.com/paper/2210.17432/) (2022)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Linguistically-Informed Self-Attention for Semantic Role Labeling](https://mlsanity.com/paper/1804.08199/) (2018)
- [GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution](https://mlsanity.com/paper/1611.04051/) (2016)
- [Empowering Diffusion Models on the Embedding Space for Text Generation](https://mlsanity.com/paper/2212.09412/) (2022)
- [Viable Dependency Parsing as Sequence Labeling](https://mlsanity.com/paper/1902.10505/) (2019)
- [Constituent Parsing as Sequence Labeling](https://mlsanity.com/paper/1810.08994/) (2018)
- [Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation](https://mlsanity.com/paper/2305.04044/) (2023)
- [Tetra-Tagging: Word-Synchronous Parsing with Linear-Time Inference](https://mlsanity.com/paper/1904.09745/) (2019)
- [HiNER: A Large Hindi Named Entity Recognition Dataset](https://mlsanity.com/paper/2204.13743/) (2022)
- [Discontinuous Constituent Parsing as Sequence Labeling](https://mlsanity.com/paper/2010.00633/) (2020)
- [4 and 7-bit Labeling for Projective and Non-Projective Dependency Trees](https://mlsanity.com/paper/2310.14319/) (2023)
- [A Diffusion Weighted Graph Framework for New Intent Discovery](https://mlsanity.com/paper/2310.15836/) (2023)
- [Inherent Dependency Displacement Bias of Transition-Based Algorithms](https://mlsanity.com/paper/2003.14282/) (2020)
- [MolGraph-xLSTM: A graph-based dual-level xLSTM framework with multi-head mixture-of-experts for enhanced molecular representation and interpretability](https://mlsanity.com/paper/2501.18439/) (2025)
- [On the Challenges of Fully Incremental Neural Dependency Parsing](https://mlsanity.com/paper/2309.16254/) (2023)
- [Dependency Graph Parsing as Sequence Labeling](https://mlsanity.com/paper/2410.17972/) (2024)
- [A Study on the Performance of U-Net Modifications in Retroperitoneal Tumor Segmentation](https://mlsanity.com/paper/2502.00314/) (2025)
- [From Partial to Strictly Incremental Constituent Parsing](https://mlsanity.com/paper/2402.02782/) (2024)

## Built on by (most cited first, 1 of 1)

- [Toward Federated Large Language Models in Medicine: A Parameter-Efficient Framework for Privacy-Preserving, Multi-Institutional Adaptation](https://mlsanity.com/paper/2601.22124/) (2026)

## Abstract (arXiv)

Pretrained Transformer encoders are the dominant approach to sequence labeling. While some alternative architectures-such as xLSTMs, structured state-space models, diffusion models, and adversarial learning-have shown promise in language modeling, few have been applied to sequence labeling, and mostly on flat or simplified tasks. We study how these architectures adapt across tagging tasks that vary in structural complexity, label space, and token dependencies, with evaluation spanning multiple languages. We find that the strong performance previously observed in simpler settings does not always generalize well across languages or datasets, nor does it extend to more complex structured tasks.