mlsanityData to 7 Oct 2026

Beyond Semantic Manipulation: Token-Space Attacks on Reward Models

arXiv 2604.02686 ↗First posted 3 Apr 2026cs.LG cs.AIcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Reward models (RMs) are widely used as optimization targets in reinforcement learning from human feedback (RLHF), yet they remain vulnerable to reward hacking. Existing attacks mainly operate within the semantic space, constructing human-readable adversarial outputs that exploit RM biases. In this work, we introduce a fundamentally different paradigm: Token Mapping Perturbation Attack (TOMPA), a framework that performs adversarial optimization directly in token space. By bypassing the standard decode-re-tokenize interface between the policy and the reward model, TOMPA enables the attack policy to optimize over raw token sequences rather than coherent natural language. Using only black-box scalar feedback, TOMPA automatically discovers non-linguistic token patterns that elicit extremely high rewards across multiple state-of-the-art RMs. Specifically, when targeting Skywork-Reward-V2-Llama-3.1-8B, TOMPA nearly doubles the reward of GPT-5 reference answers and outperforms them on 98.0% of prompts. Despite these high scores, the generated outputs degenerate into nonsensical text, revealing that RMs can be systematically exploited beyond the semantic regime and exposing a critical vulnerability in current RLHF pipelines.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 31 of its 33 references that are in the map. First posted on arXiv 3 Apr 2026.

Builds on

31 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Adversarial Training of Reward Models8 Apr

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  10. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
18 more
  1. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  2. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  3. 2021Unsolved Problems in ML Safetycited by 291
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  5. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  6. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  7. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  8. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  9. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  10. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  11. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  12. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  13. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  14. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  15. 2025One Token to Fool LLM-as-a-Judgecited by 56
  16. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  17. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
  18. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safetycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewardscited by 0
  2. 2026No Free Checker: A Survey of Verifiers for Robot Policiescited by 0
  3. 2026STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
  2. 2026Reward Stealing Attack on Large Language Models
  3. 2026Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models
  4. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  5. 2024Jailbreaking as a Reward Misspecification Problem
  6. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
  7. 2026Multimodal Reward Hacking in Reinforcement Learning
  8. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHF