mlsanityData to 14 Oct 2026

LLM Hypnosis: Exploiting User Feedback for Unauthorized Knowledge Injection to All Users

arXiv 2507.02850 ↗First posted 3 Jul 2025cs.CL cs.CR cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

We describe a vulnerability in language models (LMs) trained with user feedback, whereby a single user can persistently alter LM knowledge and behavior given only the ability to provide prompts and upvote / downvote feedback on LM outputs. To implement the attack, the attacker prompts the LM to stochastically output either a "poisoned" or benign response, then upvotes the poisoned response or downvotes the benign one. When feedback signals are used in a subsequent preference tuning behavior, LMs exhibit increased probability of producing poisoned responses even in contexts without malicious prompts. We show that this attack can be used to (1) insert factual knowledge the model did not previously possess, (2) modify code generation patterns in ways that introduce exploitable security flaws, and (3) inject fake financial news. Our finding both identifies a new qualitative feature of language model preference tuning (showing that it even highly restricted forms of preference data can be used to exert fine-grained control over behavior), and a new attack mechanism for LMs trained with user feedback (extending work on pretraining-time data poisoning and deployment-time prompt injection).

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 28 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

22 of 28 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025LLM Misalignment via Adversarial RLHF Platforms4 Mar
  2. 2024Is poisoning a real threat to LLM alignment? Maybe more so than you think17 Jun
  3. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data8 Apr
  4. 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
  5. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models16 Nov

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  7. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  8. 2023Poisoning Language Models During Instruction Tuningcited by 269
  9. 2024Stealing Part of a Production Language Modelcited by 159
  10. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  11. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  12. 2024Persistent Pre-Training Poisoning of LLMscited by 48
5 more
  1. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  2. 2024Logits of API-Protected LLMs Leak Proprietary Informationcited by 40
  3. 2024BadMerging: Backdoor Attacks Against Model Mergingcited by 34
  4. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 33
  5. 2024More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthinesscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Rewardcited by 3
  2. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs
  2. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection
  3. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection
  4. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models
  5. 2025Revisiting Backdoor Attacks on LLMs: A Stealthy and Practical Poisoning Framework via Harmless Inputs
  6. 2023On the Exploitability of Instruction Tuning
  7. 2025LLMs are Vulnerable to Malicious Prompts Disguised as Scientific Language
  8. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Models