mlsanityData to 7 Oct 2026

Autoregressive Structured Prediction with Language Models

arXiv 2210.14698 ↗First posted 26 Oct 2022cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

Recent years have seen a paradigm shift in NLP towards using pretrained language models ({PLM}) for a wide range of tasks. However, there are many difficult design decisions to represent structures (e.g. tagged text, coreference chains) in a way such that they can be captured by PLMs. Prior work on structured prediction with PLMs typically flattens the structured output into a sequence, which limits the quality of structural information being learned and leads to inferior performance compared to classic discriminative models. In this work, we describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs, allowing in-structure dependencies to be learned without any loss. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at, namely, named entity recognition, end-to-end relation extraction, and coreference resolution.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 26 Oct 2022.

Builds on

31 of 56 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  7. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  8. 2017Non-Autoregressive Neural Machine Translationcited by 457
  9. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  10. 2020Autoregressive Entity Retrievalcited by 333
  11. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  12. 2017End-to-end Neural Coreference Resolutioncited by 240
19 more
  1. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  2. 2018Higher-order Coreference Resolution with Coarse-to-fine Inferencecited by 150
  3. 2019Span-based Joint Entity and Relation Extraction with Transformer Pre-trainingcited by 132
  4. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  5. 2016Recurrent Neural Network Grammarscited by 109
  6. 2019A General Framework for Information Extraction using Dynamic Span Graphscited by 93
  7. 2022Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexitycited by 92
  8. 2019Coreference Resolution as Query-based Span Predictioncited by 91
  9. 2020Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoderscited by 87
  10. 2017Deep Joint Entity Disambiguation with Local Neural Attentioncited by 79
  11. 2014Grammar as a Foreign Languagecited by 70
  12. 2015Structured Prediction Energy Networkscited by 59
  13. 2015Learning to Search Better Than Your Teachercited by 50
  14. 2021Packed Levitated Marker for Entity and Relation Extractioncited by 49
  15. 2021Word-Level Coreference Resolutioncited by 40
  16. 2020Augmented Natural Language for Generative Sequence Labelingcited by 29
  17. 2018Learning Approximate Inference Networks for Structured Predictioncited by 13
  18. 2022A Structured Span Selectorcited by 3
  19. 2016Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 12; 2024: 15; 2025: 8; 2026: 5.

43 papers.

Sort:

  1. 2023BloombergGPT: A Large Language Model for Financecited by 923
  2. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  3. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  4. 2022Prompting Language Models for Linguistic Structurecited by 39
  5. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 32
  6. 2023CodeKGC: Code Language Model for Generative Knowledge Graph Constructioncited by 28
  7. 2024CodeS: Natural Language to Code Repository via Multi-Layer Sketchcited by 28
  8. 2024Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labelingcited by 24
  9. 2023Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuningcited by 21
  10. 2023Evaluating ChatGPT as a Recommender System: A Rigorous Approachcited by 20
  11. 2024Maverick: Efficient and Accurate Coreference Resolution Defying Recent Trendscited by 19
  12. 2023CAW-coref: Conjunction-Aware Word-level Coreference Resolutioncited by 10
31 more
  1. 2023Seq2seq is All You Need for Coreference Resolutioncited by 10
  2. 2023Informed Named Entity Recognition Decoding for Generative Language Modelscited by 6
  3. 2024Deep Autoregressive Models as Causal Inference Enginescited by 6
  4. 2024CorPipe at CRAC 2024: Predicting Zero Mentions from Raw Textcited by 6
  5. 2023PAC Prediction Sets for Large Language Models of Codecited by 5
  6. 2024An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extractioncited by 5
  7. 2024A Controlled Reevaluation of Coreference Resolution Modelscited by 4
  8. 2024Multilingual Coreference Resolution in Low-resource South Asian Languagescited by 2
  9. 2024GraphER: A Structure-aware Text-to-Graph Model for Entity and Relation Extractioncited by 2
  10. 2024Transforming Dutch: Debiasing Dutch Coreference Resolution Systems for Non-binary Pronounscited by 2
  11. 2024CoNLL#: Fine-grained Error Analysis and a Corrected Test Set for CoNLL-03 Englishcited by 2
  12. 2025Predicting Through Generation: Why Generation Is Better for Predictioncited by 2
  13. 2025Trusted Knowledge Extraction for Operations and Maintenance Intelligencecited by 2
  14. 2025CorPipe at CRAC 2025: Evaluating Multilingual Encoders for Multilingual Coreference Resolutioncited by 2
  15. 2022How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling?cited by 1
  16. 2024Consistent Joint Decision-Making with Heterogeneous Learning Modelscited by 1
  17. 2025Mapping the Course for Prompt-based Structured Predictioncited by 1
  18. 2025End-to-End Argument Mining through Autoregressive Argumentative Structure Predictioncited by 1
  19. 2023A Few-Shot Learning Focused Survey on Recent Named Entity Recognition and Relation Classification Methodscited by 0
  20. 2023Utilizing Multiple Inputs Autoregressive Models for Bearing Remaining Useful Life Predictioncited by 0
  21. 2024A Two Dimensional Feature Engineering Method for Relation Extractioncited by 0
  22. 2024ThaiCoref: Thai Coreference Resolution Datasetcited by 0
  23. 2024Exploration of Marker-Based Approaches in Argument Mining through Augmented Natural Languagecited by 0
  24. 2025Dependency Parsing is More Parameter-Efficient with Normalizationcited by 0
  25. 2025CorefInst: Leveraging LLMs for Multilingual Coreference Resolutioncited by 0
  26. 2025Efficient Seq2seq Coreference Resolution Using Entity Representationscited by 0
  27. 2026Relation Extraction Capabilities of LLMs on Clinical Text: A Bilingual Evaluation for English and Turkishcited by 0
  28. 2026Uncertainty Quantification for Named Entity Recognition via Full-Sequence and Subsequence Conformal Predictioncited by 0
  29. 2026LLMs Underperform Graph-Based Parsers on Supervised Relation Extraction for Complex Graphscited by 0
  30. 2026CorPipe at CRAC 2026: Empty Nodes and Cross-Lingual Transfer in Multilingual Coreference Resolutioncited by 0
  31. 2026LLM-INSTRUCT at UZH Shared Task 2026: Constraint-Aware Retrieval and Selective Debate for Paragraph-Level Argument Miningcited by 0