mlsanityData to 7 Oct 2026

Unitary Multi-Margin BERT for Robust Natural Language Processing

arXiv 2410.12759 ↗First posted 16 Oct 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent developments in adversarial attacks on deep learning leave many mission-critical natural language processing (NLP) systems at risk of exploitation. To address the lack of computationally efficient adversarial defense methods, this paper reports a novel, universal technique that drastically improves the robustness of Bidirectional Encoder Representations from Transformers (BERT) by combining the unitary weights with the multi-margin loss. We discover that the marriage of these two simple ideas amplifies the protection against malicious interference. Our model, the unitary multi-margin BERT (UniBERT), boosts post-attack classification accuracies significantly by 5.3% to 73.8% while maintaining competitive pre-attack accuracies. Furthermore, the pre-attack and post-attack accuracy tradeoff can be adjusted via a single scalar parameter to best fit the design requirements for the target applications.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 29 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

16 of 29 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  9. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  10. 2016Counter-fitting Word Vectors to Linguistic Constraintscited by 138
  11. 2017Deep Text Classification Can be Fooledcited by 105
  12. 2020InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspectivecited by 92
4 more
  1. 2020Better Robustness by More Coverage: Adversarial Training with Mixup Augmentation for Robust Fine-tuningcited by 39
  2. 2018Max-Mahalanobis Linear Discriminant Analysis Networkscited by 18
  3. 2021Deep Unitary Convolutional Neural Networkscited by 2
  4. 2022On the Relationship Between Adversarial Robustness and Decision Region in Deep Neural Networkscited by 1

Built on by

Not cited yet.