mlsanityData to 14 Oct 2026

Reward Stealing Attack on Large Language Models

arXiv 2610.06670 ↗First posted 5 Oct 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks also fits: Reward hacking and misspecification

also in 1 more
Abstract (arXiv)

Adversarial attacks on Large Language Models (LLMs) aim to induce harmful content. However, existing methods suffer from high computational costs or strict model-pairing dependencies, limiting their scalability and transferability. We propose Reward Stealing Attack (ReSA), an adversarial attack framework that targets the latent safety reward underlying LLM alignment. ReSA employs maximum entropy inverse reinforcement learning to recover a proxy reward model solely from the aligned model's behavior. The extracted reward is then reversed at inference time to derive an adversarial policy, efficiently implemented via a reward-guided decoding mechanism. Experiments demonstrate that a single recovered reward generalizes across prompts and diverse models to reveal a fundamental alignment vulnerability, enabling ReSA to significantly outperform existing attacks in effectiveness and transferability. The code is available at https://github.com/GarminQ/ReSA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 40 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

18 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025HarmRLVR: Weaponizing Verifiable Rewards for Harmful LLM Alignment17 Oct
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  7. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
  8. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  9. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  10. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 80
  11. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
  12. 2024The Ethics of Interaction: Mitigating Security Threats in LLMscited by 33
1 more
  1. 2025Confidence Elicitation: A New Attack Vector for Large Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective
  2. 2025Adversarial Training of Reward Models
  3. 2024Jailbreaking as a Reward Misspecification Problem
  4. 2024Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models?
  5. 2025LLM Misalignment via Adversarial RLHF Platforms
  6. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  7. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Models
  8. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models