mlsanityData to 14 Oct 2026

SenTest: Evaluating Robustness of Sentence Encoders

arXiv 2311.17722 ↗First posted 29 Nov 2023cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Contrastive learning has proven to be an effective method for pre-training models using weakly labeled data in the vision domain. Sentence transformers are the NLP counterparts to this architecture, and have been growing in popularity due to their rich and effective sentence representations. Having effective sentence representations is paramount in multiple tasks, such as information retrieval, retrieval augmented generation (RAG), and sentence comparison. Keeping in mind the deployability factor of transformers, evaluating the robustness of sentence transformers is of utmost importance. This work focuses on evaluating the robustness of the sentence encoders. We employ several adversarial attacks to evaluate its robustness. This system uses character-level attacks in the form of random character substitution, word-level attacks in the form of synonym replacement, and sentence-level attacks in the form of intra-sentence word order shuffling. The results of the experiments strongly undermine the robustness of sentence encoders. The models produce significantly different predictions as well as embeddings on perturbed datasets. The accuracy of the models can fall up to 15 percent on perturbed datasets as compared to unperturbed datasets. Furthermore, the experiments demonstrate that these embeddings does capture the semantic and syntactic structure (sentence order) of sentences. However, existing supervised classification strategies fail to leverage this information, and merely function as n-gram detectors.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

29 of 39 references.

References · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  9. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  10. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 491
  11. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 381
  12. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
17 more
  1. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  2. 2017Synthetic and Natural Noise Both Break Neural Machine Translationcited by 306
  3. 2017Generating Natural Adversarial Examplescited by 183
  4. 2019Text Processing Like Humans Do: Visually Attacking and Shielding NLP Systemscited by 101
  5. 2018On Adversarial Examples for Character-Level Neural Machine Translationcited by 85
  6. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 57
  7. 2021Model Extraction and Adversarial Transferability, Your BERT is Vulnerable!cited by 53
  8. 2020Universal Adversarial Attacks with Natural Triggers for Text Classificationcited by 51
  9. 2020CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial Text Generationcited by 44
  10. 2021Using Adversarial Attacks to Reveal the Statistical Bias in Machine Reading Comprehension Modelscited by 29
  11. 2021Grey-box Adversarial Attack And Defence For Sentiment Classificationcited by 28
  12. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  13. 2021Multi-granularity Textual Adversarial Attack with Behavior Cloningcited by 21
  14. 2020On the Robustness of Language Encoders against Grammatical Errorscited by 19
  15. 2019White-to-Black: Efficient Distillation of Black-Box Adversarial Attackscited by 14
  16. 2021ShufText: A Simple Black Box Approach to Evaluate the Fragility of Text Classification Modelscited by 5
  17. 2021BBAEG: Towards BERT-based Biomedical Adversarial Example Generation for Text Classificationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Robustness of Large Language Models Against Adversarial Attacks
  2. 2025Bridging Robustness and Generalization Against Word Substitution Attacks in NLP via the Growth Bound Matrix Approach
  3. 2026Unifying Adversarial Robustness and Training Across Text Scoring Models
  4. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts
  5. 2025RobustMask: Certified Robustness against Adversarial Neural Ranking Attack via Randomized Masking
  6. 2025Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks
  7. 2022Apple of Sodom: Hidden Backdoors in Superior Sentence Embeddings via Contrastive Learning
  8. 2025Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders