mlsanityData to 14 Oct 2026

StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

arXiv 1908.04577 ↗First posted 13 Aug 2019cs.CLcited by 109

Not on the map. Why?

Abstract (arXiv)

Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

Key facts. Cited by 109 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 36; 2022: 25; 2023: 25; 2024: 12; 2025: 8; 2026: 3.

109 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  2. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  3. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  4. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  5. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  6. 2021M6: A Chinese Multimodal Pretrainercited by 91
  7. 2021StructuralLM: Structural Pre-training for Form Understandingcited by 91
  8. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 85
  9. 2023On the (In)Effectiveness of Large Language Models for Chinese Text Correctioncited by 54
  10. 2021A Survey on Green Deep Learningcited by 52
  11. 2021HerBERT: Efficiently Pretrained Transformer-based Language Model for Polishcited by 51
  12. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 41
38 more
  1. 2022MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correctioncited by 33
  2. 2022Attention Mechanism in Neural Networks: Where it Comes and Where it Goescited by 29
  3. 2021Are Pre-trained Convolutions Better than Pre-trained Transformers?cited by 27
  4. 2022Linguistic Rules-Based Corpus Generation for Native Chinese Grammatical Error Correctioncited by 25
  5. 2024Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challengescited by 20
  6. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levelscited by 19
  7. 2021ICDAR 2021 Competition on Document VisualQuestion Answeringcited by 19
  8. 2022Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuningcited by 19
  9. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19
  10. 2021Monolingual and Cross-Lingual Acceptability Judgments with the Italian CoLA corpuscited by 18
  11. 2022Towards Unified Prompt Tuning for Few-shot Text Classificationcited by 18
  12. 2022Tele-Knowledge Pre-training for Fault Analysiscited by 18
  13. 2021Frustratingly Simple Pretraining Alternatives to Masked Language Modelingcited by 17
  14. 2023FACTIFY3M: A Benchmark for Multimodal Fact Verification with Explainability through 5W Question-Answeringcited by 16
  15. 2021DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understandingcited by 15
  16. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  17. 2023Improving Long Document Topic Segmentation Models With Enhanced Coherence Modelingcited by 15
  18. 2021Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Modelscited by 14
  19. 2022FCGEC: Fine-Grained Corpus for Chinese Grammatical Error Correctioncited by 13
  20. 2022Paraphrase Identification with Deep Learning: A Review of Datasets and Methodscited by 13
  21. 2023Correct Like Humans: Progressive Learning Framework for Chinese Text Error Correctioncited by 13
  22. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 12
  23. 2023MGeo: Multi-Modal Geographic Pre-Training Methodcited by 12
  24. 2023WYWEB: A NLP Evaluation Benchmark For Classical Chinesecited by 12
  25. 2021How to Query Language Models?cited by 11
  26. 2021Structure Inducing Pre-Trainingcited by 10
  27. 2021Shaking Syntactic Trees on the Sesame Street: Multilingual Probing with Controllable Perturbationscited by 10
  28. 2022Incorporating Explicit Knowledge in Pre-trained Language Models for Passage Re-rankingcited by 10
  29. 2023QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Searchcited by 10
  30. 2022CGoDial: A Large-Scale Benchmark for Chinese Goal-oriented Dialog Evaluationcited by 9
  31. 2023MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Searchcited by 9
  32. 2023Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reportscited by 9
  33. 2024Rethinking the Roles of Large Language Models in Chinese Grammatical Error Correctioncited by 9
  34. 2021AliMe MKG: A Multi-modal Knowledge Graph for Live-streaming E-commercecited by 8
  35. 2022BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representationcited by 8
  36. 2021Pre-training Universal Language Representationcited by 7
  37. 2022Paragraph-based Transformer Pre-training for Multi-Sentence Inferencecited by 7
  38. 2023Zero-Shot Text Classification via Self-Supervised Tuningcited by 6

50 most cited of 109 All 109 (JSON)