mlsanityData to 7 Oct 2026

Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples

arXiv 2209.02128 ↗First posted 5 Sep 2022cs.CLcited by 65

Not on the map. Why?

Cited by 27 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in the development of large language models have resulted in public access to state-of-the-art pre-trained language models (PLMs), including Generative Pre-trained Transformer 3 (GPT-3) and Bidirectional Encoder Representations from Transformers (BERT). However, evaluations of PLMs, in practice, have shown their susceptibility to adversarial attacks during the training and fine-tuning stages of development. Such attacks can result in erroneous outputs, model-generated hate speech, and the exposure of users' sensitive information. While existing research has focused on adversarial attacks during either the training or the fine-tuning of PLMs, there is a deficit of information on attacks made between these two development phases. In this work, we highlight a major security vulnerability in the public release of GPT-3 and further investigate this vulnerability in other state-of-the-art PLMs. We restrict our work to pre-trained models that have not undergone fine-tuning. Further, we underscore token distance-minimized perturbations as an effective adversarial approach, bypassing both supervised and unsupervised quality measures. Following this approach, we observe a significant decrease in text classification quality when evaluating for semantic similarity.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 5 Sep 2022.

Builds on

32 of 43 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  11. 2019How Can We Know What Language Models Know?cited by 953
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
20 more
  1. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  2. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  3. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  4. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  5. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  6. 2018Generating Natural Language Adversarial Examplescited by 422
  7. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  8. 2018Generating Adversarial Examples with Adversarial Networkscited by 256
  9. 2019Dice Loss for Data-imbalanced NLP Taskscited by 180
  10. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  11. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  12. 2022Should You Mask 15% in Masked Language Modeling?cited by 117
  13. 2019On the Impact of the Activation Function on Deep Neural Networks Trainingcited by 101
  14. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  15. 2021bert2BERT: Towards Reusable Pretrained Language Modelscited by 70
  16. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  17. 2017Adversarial examples for generative modelscited by 58
  18. 2021Context-aware Adversarial Training for Name Regularity Bias in Named Entity Recognitioncited by 19
  19. 2020Syntactic Structure Distillation Pretraining For Bidirectional Encoderscited by 18
  20. 2020Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 24; 2025: 21; 2026: 9.

65 papers.

Sort:

By area:

  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  5. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  6. 2022Prompting GPT-3 To Be Reliablecited by 258
  7. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  8. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  9. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  10. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  11. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  12. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
38 more
  1. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  2. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  3. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  4. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  5. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  6. 2024From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agentscited by 76
  7. 2023LLM Censorship: A Machine Learning Challenge or a Computer Security Problem?cited by 47
  8. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46
  9. 2023Can LLMs Follow Simple Rules?cited by 45
  10. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applicationscited by 42
  11. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  12. 2025A Closer Look at System Prompt Robustnesscited by 31
  13. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  14. 2025We Can't Understand AI Using our Existing Vocabularycited by 25
  15. 2024Malla: Demystifying Real-world Large Language Model Integrated Malicious Servicescited by 24
  16. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20
  17. 2022Despite "super-human" performance, current LLMs are unsuited for decisions about ethics and safetycited by 13
  18. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 13
  19. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  20. 2025Tool Preferences in Agentic LLMs are Unreliablecited by 13
  21. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  22. 2025Prompt Injection 2.0: Hybrid AI Threatscited by 12
  23. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 11
  24. 2024PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Modelscited by 7
  25. 2024Exfiltration of personal information from ChatGPT via prompt injectioncited by 7
  26. 2024Applying Pre-trained Multilingual BERT in Embeddings for Improved Malicious Prompt Injection Attacks Detectioncited by 7
  27. 2025StruPhantom: Evolutionary Injection Attacks on Black-Box Tabular Agents Powered by Large Language Modelscited by 6
  28. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  29. 2024ProTransformer: Robustify Transformers via Plug-and-Play Paradigmcited by 5
  30. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 5
  31. 2024Generative AI Security: Challenges and Countermeasurescited by 4
  32. 2024Non-Halting Queries: Exploiting Fixed Points in LLMscited by 4
  33. 2025When Harry Meets Superman: The Role of The Interlocutor in Persona-Based Dialogue Generationcited by 4
  34. 2026Agents at Risk: How Users Unwittingly Undermine LLM Safetycited by 3
  35. 2023Creating Large Language Model Resistant Exams: Guidelines and Strategiescited by 2
  36. 2025Playing Devil's Advocate: Unmasking Toxicity and Vulnerabilities in Large Vision-Language Modelscited by 2
  37. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  38. 2025AI Security Beyond Core Domains: Resume Screening as a Case Study of Adversarial Vulnerabilities in Specialized LLM Applicationscited by 2

50 most cited of 65 All 65 (JSON)