mlsanityData to 14 Oct 2026

Keep on Swimming: Real Attackers Only Need Partial Knowledge of a Multi-Model System

arXiv 2410.23483 ↗First posted 30 Oct 2024cs.LG cs.AI cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent approaches in machine learning often solve a task using a composition of multiple models or agentic architectures. When targeting a composed system with adversarial attacks, it might not be computationally or informationally feasible to train an end-to-end proxy model or a proxy model for every component of the system. We introduce a method to craft an adversarial attack against the overall multi-model system when we only have a proxy model for the final black-box model, and when the transformation applied by the initial models can make the adversarial perturbations ineffective. Current methods handle this by applying many copies of the first model/transformation to an input and then re-use a standard adversarial attack by averaging gradients, or learning a proxy model for both stages. To our knowledge, this is the first attack specifically designed for this threat model and our method has a substantially higher attack success rate (80% vs 25%) and contains 9.4% smaller perturbations (MSE) compared to prior state-of-the-art methods. Our experiments focus on a supervised image pipeline, but we are confident the attack will generalize to other multi-model settings [e.g. a mix of open/closed source foundation models], or agentic systems

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 37 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

31 of 37 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  11. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  12. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
19 more
  1. 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
  2. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 501
  3. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 446
  4. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  5. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  6. 2021TrOCR: Transformer-based Optical Character Recognition with Pre-trained Modelscited by 279
  7. 2019Character Region Awareness for Text Detectioncited by 278
  8. 2024Stealing Part of a Production Language Modelcited by 159
  9. 2019Stateful Detection of Black-Box Adversarial Attackscited by 56
  10. 2022Practical Evaluation of Adversarial Robustness via Adaptive Auto Attackcited by 44
  11. 2018Fairness Under Compositioncited by 43
  12. 2024From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Modelscited by 43
  13. 2020Individual Fairness in Pipelinescited by 22
  14. 2017Fair Pipelinescited by 21
  15. 2024Grounding and Evaluation for Large Language Models: Practical Challenges and Lessons Learned (Survey)cited by 20
  16. 2023I See Dead People: Gray-Box Adversarial Attack on Image-To-Text Modelscited by 17
  17. 2021Adversarial Imaging Pipelinescited by 15
  18. 2022Multi Stage Screening: Enforcing Fairness and Maximizing Efficiency in a Pre-Existing Pipelinecited by 8
  19. 2023Sequential Strategic Screeningcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Learn from the Past: A Proxy Guided Adversarial Defense Framework with Self Distillation Regularization
  2. 2025Adversarial Agents: Black-Box Evasion Attacks with Reinforcement Learning
  3. 2022Threat Model-Agnostic Adversarial Defense using Diffusion Models
  4. 2024A Novel Approach to Guard from Adversarial Attacks using Stable Diffusion
  5. 2024Hijacking Attacks against Neural Networks by Analyzing Training Data
  6. 2025Embedding Hidden Adversarial Capabilities in Pre-Trained Diffusion Models
  7. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attacks
  8. 2026Adversarial Diffusion Across Modalities: A Fusion Survey of Attacks, Defenses, and Evaluation for Text, Vision, and Vision-Language Models