mlsanityData to 7 Oct 2026

UniBERT: Adversarial Training for Language-Universal Representations

arXiv 2503.12608 ↗First posted 16 Mar 2025cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper presents UniBERT, a compact multilingual language model that uses an innovative training framework that integrates three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on a meticulously curated Wikipedia corpus spanning 107 languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks. Comprehensive evaluations on four tasks - named entity recognition, natural language inference, question answering, and semantic textual similarity - demonstrate that our multilingual training strategy enhanced by an adversarial objective significantly improves cross-lingual generalization. Specifically, UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.17%, and statistical analysis confirms the significance of these gains (p-value = 0.0181). This work highlights the benefits of combining adversarial training and knowledge distillation to build scalable and robust language models, thus advancing the field of multilingual and cross-lingual natural language processing.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 16 Mar 2025.

Builds on

39 of 58 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
27 more
  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  6. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  7. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  8. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  9. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  10. 2023DataDAM: Efficient Dataset Distillation with Attention Matchingcited by 83
  11. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  12. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 70
  13. 2016Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classificationcited by 48
  14. 2021Domain Adversarial Neural Networks for Domain Generalization: When It Works and How to Improvecited by 46
  15. 2020The birth of Romanian BERTcited by 44
  16. 2022English Contrastive Learning Can Learn Universal Cross-lingual Sentence Embeddingscited by 26
  17. 2019Low-Resource Corpus Filtering using Multilingual Sentence Embeddingscited by 25
  18. 2021Distilling the Knowledge of Romanian BERTs Using Multiple Teacherscited by 13
  19. 2023DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognitioncited by 10
  20. 2024LAMPAT: Low-Rank Adaption for Multilingual Paraphrasing Using Adversarial Trainingcited by 8
  21. 2024Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applicationscited by 7
  22. 2024HistNERo: Historical Named Entity Recognition for the Romanian Languagecited by 5
  23. 2023Romanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibitioncited by 4
  24. 2024RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generationcited by 3
  25. 2023Multilingual Multiword Expression Identification Using Lateral Inhibition and Domain Adaptationcited by 2
  26. 2022EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learningcited by 1
  27. 2022TA-DA: Topic-Aware Domain Adaptation for Scientific Keyphrase Identification and Classification (Student Abstract)cited by 1

Built on by

Not cited yet.