mlsanityData to 7 Oct 2026

Unifying Adversarial Robustness and Training Across Text Scoring Models

arXiv 2602.00857 ↗First posted 31 Jan 2026cs.CL cs.IRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Research on adversarial robustness in language models is currently fragmented across applications and attacks, obscuring shared vulnerabilities. In this work, we propose unifying the study of adversarial robustness in text scoring models spanning dense retrievers, rerankers, and reward models. This motivates adapting both attacks and adversarial training methods across model roles. Unlike open-ended generation, text scoring failures are directly testable: an attack succeeds when an irrelevant or rejected text outscores a relevant or chosen one. Using this principled lens of text scoring, we demonstrate that current adversarial training formulations for language models are often short-sighted, failing to effectively generalize across attacks. To address this, we introduce multiple adversarial training methods for text scoring models and show that combining complementary training methods can yield strong robustness while also improving task effectiveness. We also highlight the practical value of our approach for RLHF, showing that our adversarially trained reward models mitigate reward hacking and support the training of better-aligned LLMs. We provide our code and models for further study.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 67 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

52 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 51

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2013Intriguing properties of neural networkscited by 5,114
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
38 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  8. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  9. 2020Fact or Fiction: Verifying Scientific Claimscited by 546
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  11. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  13. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  14. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  15. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  16. 2020Overview of the TREC 2019 deep learning trackcited by 282
  17. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  18. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  19. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  20. 2020CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claimscited by 195
  21. 2021Overview of the TREC 2020 deep learning trackcited by 193
  22. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  23. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  24. 2020Adversarial Training for Large Neural Language Modelscited by 109
  25. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  26. 2024How to Evaluate Reward Models for RLHFcited by 84
  27. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  28. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  29. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  30. 2022PRADA: Practical Black-Box Adversarial Attacks against Neural Ranking Modelscited by 44
  31. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  32. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
  33. 2025Adversarial Training of Reward Modelscited by 17
  34. 2025Hard Negatives, Hard Lessons: Revisiting Training Data Quality for Robust Information Retrieval with LLMscited by 17
  35. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimizationcited by 14
  36. 2023A Study on FGSM Adversarial Training for Neural Retrievalcited by 10
  37. 2025MixAT: Combining Continuous and Discrete Adversarial Training for LLMscited by 10
  38. 2023Perturbation-Invariant Adversarial Training for Neural Ranking Models: Improving the Effectiveness-Robustness Trade-Offcited by 7

50 most cited of 51 All 51 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Tougher Text, Smarter Models: Raising the Bar for Adversarial Defence Benchmarks
  2. 2025Selective Adversarial Attacks on LLM Benchmarks
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models
  4. 2025RobustMask: Certified Robustness against Adversarial Neural Ranking Attack via Randomized Masking
  5. 2022Rethinking Textual Adversarial Defense for Pre-trained Language Models
  6. 2026Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation
  7. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectives
  8. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks