mlsanityData to 14 Oct 2026

VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understanding

arXiv 2211.00430 ↗First posted 1 Nov 2022cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models have achieved promising performance on general benchmarks, but underperform when migrated to a specific domain. Recent works perform pre-training from scratch or continual pre-training on domain corpora. However, in many specific domains, the limited corpus can hardly support obtaining precise representations. To address this issue, we propose a novel Transformer-based language model named VarMAE for domain-adaptive language understanding. Under the masked autoencoding objective, we design a context uncertainty learning module to encode the token's context into a smooth latent distribution. The module can produce diverse and well-formed contextual representations. Experiments on science- and finance-domain NLU tasks demonstrate that VarMAE can be efficiently adapted to new domains with limited resources.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 1 Nov 2022.

Builds on

43 of 69 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2013Auto-Encoding Variational Bayescited by 4,053
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
31 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2018Deep contextualized word representationscited by 2,290
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  6. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  7. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 834
  9. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  10. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  11. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
  12. 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
  13. 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 278
  14. 2019Structural Scaffolds for Citation Intent Classification in Scientific Publicationscited by 130
  15. 2018Construction of the Literature Graph in Semantic Scholarcited by 118
  16. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  17. 2020LEGAL-BERT: The Muppets straight out of Law Schoolcited by 111
  18. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 109
  19. 2018A Corpus with Multi-Level Annotations of Patients, Interventions and Outcomes to Support Language Processing for Medical Literaturecited by 79
  20. 2020ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learningcited by 72
  21. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  22. 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
  23. 2019ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representationscited by 35
  24. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  25. 2019A Surprisingly Effective Fix for Deep Latent Variable Modeling of Textcited by 32
  26. 2020Multi-Stage Pre-training for Low-Resource Domain Adaptationcited by 24
  27. 2020A Batch Normalized Inference Network Keeps the KL Vanishing Awaycited by 20
  28. 2020SLK-NER: Exploiting Second-order Lexicon Knowledge for Chinese NERcited by 7
  29. 2020An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-trainingcited by 7
  30. 2020Hierarchical Interaction Networks with Rethinking Mechanism for Document-level Sentiment Analysiscited by 5
  31. 2022PALI-NLP at SemEval-2022 Task 4: Discriminative Fine-tuning of Transformers for Patronizing and Condescending Language Detectioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 3; 2025: 3; 2026: 1.

12 papers.

  1. 2023Supervised Adversarial Contrastive Learning for Emotion Recognition in Conversationscited by 19
  2. 2023Explore the Potential of LLMs in Misinformation Detection: An Empirical Studycited by 12
  3. 2024Exploring the use of a Large Language Model for data extraction in systematic reviews: a rapid feasibility studycited by 8
  4. 2025Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Paperscited by 5
  5. 2023Structured Probabilistic Codingcited by 3
  6. 2023VOLTA: Improving Generative Diversity by Variational Mutual Information Maximizing Autoencodercited by 2
  7. 2024Representation Learning with Conditional Information Flow Maximizationcited by 2
  8. 2024Structural Entropy Guided Probabilistic Codingcited by 1
  9. 2025An Information-theoretic Multi-task Representation Learning Framework for Natural Language Understandingcited by 1
  10. 2023UCAS-IIE-NLP at SemEval-2023 Task 12: Enhancing Generalization of Multilingual BERT for Low-resource Sentiment Analysiscited by 0
  11. 2025Leveraging GANs for citation intent classification and its impact on citation network analysiscited by 0
  12. 2026Predictor-Guided Latent Space Codon Optimization for Maximizing Protein Expressioncited by 0