# Specification Self-Correction: Mitigating In-Context Reward Hacking Through Test-Time Refinement

arXiv: https://arxiv.org/abs/2507.18742 · HTML: https://mlsanity.com/paper/2507.18742/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 11 of its 12 references that are in the map.
- First posted on arXiv 24 Jul 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)

## Builds on (most cited first, 11 of 11)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models](https://mlsanity.com/paper/2201.03544/) (2022)
- [Defining and Characterizing Reward Hacking](https://mlsanity.com/paper/2209.13085/) (2022)
- [Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models](https://mlsanity.com/paper/2406.10162/) (2024)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [A Theoretical Understanding of Self-Correction through In-context Alignment](https://mlsanity.com/paper/2405.18634/) (2024)
- [Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack](https://mlsanity.com/paper/2410.06491/) (2024)
- [MetaSC: Test-Time Safety Specification Optimization for Language Models](https://mlsanity.com/paper/2502.07985/) (2025)

## Built on by (most cited first, 6 of 6)

- [Adaptation of Agentic AI: A Survey of Post-Training, Memory, and Skills](https://mlsanity.com/paper/2512.16301/) (2025)
- [Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis](https://mlsanity.com/paper/2601.20103/) (2026)
- [Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents](https://mlsanity.com/paper/2607.24300/) (2026)
- [Beyond Scalar Rewards: Dense Feedback for LLM Policy Synthesis in Sequential Social Dilemmas](https://mlsanity.com/paper/2603.19453/) (2026)
- [Distilling Feedback into Memory-as-a-Tool](https://mlsanity.com/paper/2601.05960/) (2026)
- [Discovering Agentic Safety Specifications from 1-Bit Danger Signals](https://mlsanity.com/paper/2604.23210/) (2026)

## Abstract (arXiv)

Language models (LMs) are susceptible to in-context reward hacking, where they exploit flaws in tainted or faulty written specifications or rubrics to achieve high scores without fulfilling the user's true intent. We introduce Specification Self-Correction (SSC), a novel, test-time framework that enables an LM to identify and correct flaws within its own guiding specification. SSC employs a multi-step inference process where the model first generates a response based on a potentially tainted specification, critiques its output, and then revises the specification itself to remove the exploitable loophole. A final, more robust response is then generated using this self-corrected specification. Across experiments spanning creative writing and agentic coding tasks with several LMs, we demonstrate that while models initially game tainted specifications in 50-70% of cases, the SSC process reduces this vulnerability by over 90%. This dynamic repair occurs at inference time, requires no weight modification, and leads to more robustly aligned model behavior. Code at https://github.com/vicgalle/specification-self-correction .