mlsanityData to 14 Oct 2026

It Takes One to Bias Them All: Breaking Bad with One-Shot GRPO

arXiv 2606.10931 ↗First posted 9 Jun 2026cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Warning: This paper contains several toxic and offensive statements. Modern large language models (LLMs) are typically aligned through large-scale post-training to ensure fair and reliable behavior. In this work, we investigate how easily such guardrails can be broken by Group Relative Policy Optimization (GRPO). We show that one-shot GRPO training on a single biased example is sufficient to induce systematic bias, with stereotype-driven reasoning generalizing across attributes, categories, and benchmarks. We further find that models differ in their susceptibility based on the initial likelihood of producing biased outputs. Our results reveal a critical vulnerability in post-training: alignment can be overridden by a single example.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 67 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

49 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text30 Jan
  2. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization8 Jan
  3. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients21 Oct
  4. 2025Geometric-Mean Policy Optimization28 Jul
  5. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  6. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2024Mixtral of Expertscited by 1,823
  8. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  9. 2023PaLM 2 Technical Reportcited by 1,162
  10. 2018Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labelscited by 1,059
  11. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  12. 2025Group Sequence Policy Optimizationcited by 835
29 more
  1. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 766
  2. 2021Domain Generalization: A Surveycited by 764
  3. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  4. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  5. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  6. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  7. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 353
  8. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  9. 2023Poisoning Language Models During Instruction Tuningcited by 269
  10. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 194
  11. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  12. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  13. 2018Adversarial Attack and Defense on Graph Data: A Surveycited by 167
  14. 2019A Meta-Transfer Objective for Learning to Disentangle Causal Mechanismscited by 158
  15. 2022Interventional Causal Representation Learningcited by 142
  16. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  17. 2023On the Exploitability of Instruction Tuningcited by 116
  18. 2023Combating Misinformation in the Age of LLMs: Opportunities and Challengescited by 95
  19. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  20. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  21. 2023WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Modelscited by 42
  22. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 40
  23. 2023Label Poisoning is All You Needcited by 29
  24. 2025Data Poisoning in Deep Learning: A Surveycited by 20
  25. 2024Evaluating Gender Bias of LLMs in Making Morality Judgementscited by 15
  26. 2021Poisoning Attack against Estimating from Pairwise Comparisonscited by 14
  27. 2025Guiding LLM Decision-Making with Fairness Reward Modelscited by 9
  28. 2024Transparent and Coherent Procedural Mistake Detectioncited by 4
  29. 2025Domain Generalization Under Posterior Driftcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Mitigating Selection Bias in Large Language Models via Permutation-Aware GRPO
  2. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  3. 2026BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
  4. 2025Group Causal Policy Optimization for Post-Training Large Language Models
  5. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  6. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPO
  7. 2025Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Data
  8. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization