mlsanityData to 13 Oct 2026

Semantics-aware BERT for Language Understanding

arXiv 1909.02209 ↗First posted 5 Sep 2019cs.CLcited by 111

Not on the map. Why?

Abstract (arXiv)

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

Key facts. Cited by 111 papers in the map (data through 13 Oct 2026). First posted on arXiv 5 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 36; 2022: 24; 2023: 21; 2024: 14; 2025: 10; 2026: 6.

111 papers.

Sort:

By area:

  1. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 798
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 461
  3. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  4. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  5. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  6. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  7. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  8. 2021Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrificecited by 30
  9. 2021Supervising Model Attention with Human Explanations for Robust Natural Language Inferencecited by 29
  10. 2021Combining pre-trained language models and structured knowledgecited by 25
  11. 2021A Short Survey of Pre-trained Language Models for Conversational AI-A NewAge in NLPcited by 25
  12. 2021Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Taskscited by 22
38 more
  1. 2022DABERT: Dual Attention Enhanced BERT for Semantic Matchingcited by 21
  2. 2023Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflictscited by 21
  3. 2021Structural Pre-training for Dialogue Comprehensioncited by 20
  4. 2022Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformerscited by 20
  5. 2022E2S2: Encoding-Enhanced Sequence-to-Sequence Pretraining for Language Understanding and Generationcited by 19
  6. 2022Improving Semantic Matching through Dependency-Enhanced Pre-trained Model with Adaptive Fusioncited by 18
  7. 2021Reinforcement Learning-powered Semantic Communication via Semantic Similaritycited by 15
  8. 2023L3Cube-IndicSBERT: A simple approach for learning cross-lingual sentence representations using multilingual BERTcited by 15
  9. 2023Universal Multimodal Representation for Language Understandingcited by 14
  10. 2024Semantic Sensitivities and Inconsistent Predictions: Measuring the Fragility of NLI Modelscited by 14
  11. 2021Seeking Common but Distinguishing Difference, A Joint Aspect-based Sentiment Analysis Modelcited by 13
  12. 2021Sentence Similarity Based on Contextscited by 12
  13. 2024Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inferencecited by 11
  14. 2021Multi-turn Dialogue Reading Comprehension with Pivot Turns and Knowledgecited by 10
  15. 2024The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Modelscited by 10
  16. 2021Dialogue-oriented Pre-trainingcited by 9
  17. 2021Weakly Supervised Explainable Phrasal Reasoning with Neural Fuzzy Logiccited by 9
  18. 2024Comateformer: Combined Attention Transformer for Semantic Sentence Matchingcited by 9
  19. 2021Syntax-Aware Graph-to-Graph Transformer for Semantic Role Labellingcited by 8
  20. 2021CodemixedNLP: An Extensible and Open NLP Toolkit for Code-Mixingcited by 8
  21. 2024Grounding Natural Language to SQL Translation with Data-Based Self-Explanationscited by 8
  22. 2025StructCoh: Structured Contrastive Learning for Context-Aware Text Semantic Matchingcited by 8
  23. 2021Grammatical Error Correction as GAN-like Sequence Labelingcited by 6
  24. 2021Smoothing Dialogue States for Open Conversational Machine Readingcited by 6
  25. 2023Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizerscited by 6
  26. 2021A Temporal Variational Model for Story Generationcited by 5
  27. 2021What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Studycited by 5
  28. 2023Improving Aspect-Based Sentiment with End-to-End Semantic Role Labeling Modelcited by 5
  29. 2023N-Gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understandingcited by 5
  30. 2024GAMedX: Generative AI-based Medical Entity Data Extractor Using Large Language Modelscited by 5
  31. 2021Improving BERT Pretraining with Syntactic Supervisioncited by 4
  32. 2021ExBERT: An External Knowledge Enhanced BERT for Natural Language Inferencecited by 4
  33. 2022Unintended Bias in Language Model-driven Conversational Recommendationcited by 4
  34. 2023ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Modelscited by 4
  35. 2023Multi-turn Dialogue Comprehension from a Topic-aware Perspectivecited by 4
  36. 2021BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matchingcited by 3
  37. 2022FedQAS: Privacy-aware machine reading comprehension with federated learningcited by 3
  38. 2022Analyzing Multi-Task Learning for Abstractive Text Summarizationcited by 3

50 most cited of 111 All 111 (JSON)