mlsanityData to 7 Oct 2026

DeepStruct: Pretraining of Language Models for Structure Prediction

arXiv 2205.10475 ↗First posted 21 May 2022cs.CL cs.AI cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

We introduce a method for improving the structural understanding abilities of language models. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 55 of its 105 references that are in the map. First posted on arXiv 21 May 2022.

Builds on

55 of 105 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
38 more
  1. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  2. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 760
  3. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  4. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  5. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  6. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  7. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 271
  8. 2019Entity, Relation, and Event Extraction with Contextualized Span Representationscited by 261
  9. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  10. 2019Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systemscited by 215
  11. 2020MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselinescited by 201
  12. 2019A Unified MRC Framework for Named Entity Recognitioncited by 199
  13. 2021Structured Prediction as Translation between Augmented Natural Languagescited by 183
  14. 2020How Context Affects Language Models' Factual Predictionscited by 181
  15. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
  16. 2019Simple BERT Models for Relation Extraction and Semantic Role Labelingcited by 154
  17. 2018Higher-order Coreference Resolution with Coarse-to-fine Inferencecited by 150
  18. 2020Named Entity Recognition as Dependency Parsingcited by 150
  19. 2021A Unified Generative Framework for Various NER Subtaskscited by 141
  20. 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
  21. 2019Span-based Joint Entity and Relation Extraction with Transformer Pre-trainingcited by 132
  22. 2019BERT for Coreference Resolution: Baselines and Analysiscited by 125
  23. 2021Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Basescited by 107
  24. 2020Language Models are Open Knowledge Graphscited by 95
  25. 2019FewRel 2.0: Towards More Challenging Few-Shot Relation Classificationcited by 94
  26. 2019A General Framework for Information Extraction using Dynamic Span Graphscited by 93
  27. 2019Coreference Resolution as Query-based Span Predictioncited by 91
  28. 2019A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Fillingcited by 85
  29. 2021Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extractioncited by 73
  30. 2020Blank Language Modelscited by 43
  31. 2019Cloze-driven Pretraining of Self-attention Networkscited by 41
  32. 2019Joint Extraction of Entities and Relations Based on a Novel Decomposition Strategycited by 41
  33. 2018One for All: Neural Joint Modeling of Entities and Eventscited by 37
  34. 2019Text Infillingcited by 32
  35. 2018Large-Scale Multi-Domain Belief Tracking with Knowledge Sharingcited by 29
  36. 2020Augmented Natural Language for Generative Sequence Labelingcited by 29
  37. 2020The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understandingcited by 28
  38. 2019Dependency or Span, End-to-End Uniform Semantic Role Labelingcited by 22

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 19; 2024: 16; 2025: 4.

45 papers.

Sort:

By area:

  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  3. 2023InstructUIE: Multi-task Instruction Tuning for Unified Information Extractioncited by 167
  4. 2023Revisiting Relation Extraction in the era of Large Language Modelscited by 100
  5. 2023GPT-RE: In-context Learning for Relation Extraction using Large Language Modelscited by 62
  6. 2022Generative Knowledge Graph Construction: A Reviewcited by 49
  7. 2023Universal Information Extraction as Unified Semantic Matchingcited by 47
  8. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  9. 2023PromptNER: Prompting For Named Entity Recognitioncited by 35
  10. 2024KnowCoder: Coding Structured Knowledge into LLMs for Universal Information Extractioncited by 33
  11. 2023Revisiting Large Language Models as Zero-shot Relation Extractorscited by 29
  12. 2023CodeKGC: Code Language Model for Generative Knowledge Graph Constructioncited by 28
33 more
  1. 2023SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understandingcited by 18
  2. 2024Measuring Vision-Language STEM Skills of Neural Modelscited by 14
  3. 2024Research Trends for the Interplay between Large Language Models and Knowledge Graphscited by 14
  4. 2024Meta In-Context Learning Makes Large Language Models Better Zero and Few-Shot Relation Extractorscited by 13
  5. 2023On the Evolution of Knowledge Graphs: A Survey and Perspectivecited by 9
  6. 2024Retrieval-Augmented Generation-based Relation Extractioncited by 9
  7. 2022Benchmarking Language Models for Code Syntax Understandingcited by 8
  8. 2023JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuningcited by 6
  9. 2024PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognitioncited by 6
  10. 2024Recall, Retrieve and Reason: Towards Better In-Context Relation Extractioncited by 6
  11. 2024Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworkscited by 6
  12. 2024Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Studycited by 5
  13. 2025Event Extraction in Large Language Modelcited by 5
  14. 2023Linguistic representations for fewer-shot relation extraction across domainscited by 4
  15. 2023MSAC: Multiple Speech Attribute Control Method for Reliable Speech Emotion Recognitioncited by 4
  16. 2024BioMistral-NLU: Towards More Generalizable Medical Language Understanding through Instruction Tuningcited by 4
  17. 2025KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applicationscited by 4
  18. 2022PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completioncited by 3
  19. 2023FRACAS: A FRench Annotated Corpus of Attribution relations in newScited by 3
  20. 2022gBuilder: A Scalable Knowledge Graph Construction System for Unstructured Corpuscited by 2
  21. 2023Mapping and Cleaning Open Commonsense Knowledge Bases with Generative Translationcited by 2
  22. 2025Trusted Knowledge Extraction for Operations and Maintenance Intelligencecited by 2
  23. 2022IELM: An Open Information Extraction Benchmark for Pre-Trained Language Modelscited by 1
  24. 2023Sentence-Level Relation Extraction via Contrastive Learning with Descriptive Relation Promptscited by 1
  25. 2024A LayoutLMv3-Based Model for Enhanced Relation Extraction in Visually-Rich Documentscited by 1
  26. 2025A systematic review of relation extraction task since the emergence of Transformerscited by 1
  27. 2023Semantic Information Extraction for Text Data with Probability Graphcited by 0
  28. 2023A Few-Shot Learning Focused Survey on Recent Named Entity Recognition and Relation Classification Methodscited by 0
  29. 2024Structured Language Generation Model: Loss Calibration and Formatted Decoding for Robust Structure Prediction and Knowledge Retrievalcited by 0
  30. 2024Leveraging Linguistically Enhanced Embeddings for Open Information Extractioncited by 0
  31. 2024A Two Dimensional Feature Engineering Method for Relation Extractioncited by 0
  32. 2024Assessing the Performance of Chinese Open Source Large Language Models in Information Extraction Taskscited by 0
  33. 2024End-to-End Trainable Retrieval-Augmented Generation for Relation Extractioncited by 0