The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries
Not on the map. Why?
Abstract (arXiv)
What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? We find that commercial system prompts can override safety training, causing frontier models to lie about medical risks, dismiss safety concerns, and prioritise profit over user welfare. Testing 8 models in scenarios where commercial objectives conflict with user safety -- a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings -- we uncover catastrophic failures: models fabricating safety information, explicitly reasoning they should refuse but proceeding anyway, and actively discouraging users from consulting doctors. Most alarmingly, models show no "red line", their willingness to comply with harmful requests does not decrease as potential consequences escalate from minor to life-threatening. Our findings suggest that current safety training does not generalise to commercial deployment contexts.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 6 of its 6 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
6 of 6 references.
References · 6
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2022Red Teaming Language Models with Language Modelscited by 1,026
- 2023Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the MACHIAVELLI Benchmarkcited by 182
- 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 28
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users
- 2025From Refusal to Recovery: A Control-Theoretic Approach to Generative AI Guardrails
- 2026Agent Safety Is Action Alignment
- 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
- 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systems
- 2026Why Do AI Agents Break Rules? How Framing, Context, and Social Signals Shape Compliance
- 2026SafePro: Evaluating the Safety of Professional-Level AI Agents
- 2026The Profit Alignment Problem: How Profit Mandates Induce Alignment Failures in LLMs