mlsanityData to 14 Oct 2026

Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search

arXiv 2606.27291 ↗First posted 25 Jun 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

Abstract (arXiv)

Job-search platforms rely on low-bandwidth query interfaces that often fail to capture the high-dimensional complexity of candidate profiles. We present an end-to-end RLAIF (Reinforcement Learning from AI Feedback) framework to generate portable job search queries, terms that abstract away seeker-specific identifiers while preserving generalizable qualifications. This task introduces a highly adversarial reward surface where policy optimization frequently exploits flaws in LLM-as-judge rubrics, resulting in degenerate verbatim-copying behaviors. We conducted comprehensive empirical experiments to isolate the impact of optimization mechanics against structured reward engineering. Our results demonstrate that for critic-free optimizers, performance is overwhelmingly dictated by robust reward shaping, rendering the specific choice of algorithm largely immaterial. While critic-free per-rollout baseline methods (RLOO and REINFORCE++) natively resist reward-hacking, the group-relative advantage normalization in GRPO appears uniquely sensitive to spurious reward signals, making it disproportionately susceptible to exploitation. We show that introducing a deterministic, rule-based reward floor to correct for rewards assigned to verbatim copying mitigates this failure mode, resulting in a substantial +0.147 quality improvement on a cross-family evaluation judge. Ultimately, we show that the training-time reward model inflates performance gains by 2.4×, confirming that the training success is fundamentally dependent on enforcing reward-shaping disciplines rather than selecting alternative optimizers.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 33 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

19 of 33 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  10. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  11. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  12. 2019Document Expansion by Query Predictioncited by 231
7 more
  1. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  2. 2018Categorizing Variants of Goodhart's Lawcited by 149
  3. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  4. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  5. 2017Task-Oriented Query Reformulation with Reinforcement Learningcited by 39
  6. 2018Towards Deep and Representation Learning for Talent Search at LinkedIncited by 19
  7. 2018Talent Search and Recommendation Systems at LinkedIn: Practical Challenges and Lessons Learnedcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Reward Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Laws
  2. 2025AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
  3. 2026IG-Search: Step-Level Information Gain Rewards for Search-Augmented Reasoning
  4. 2025AgentRM: Enhancing Agent Generalization with Reward Modeling
  5. 2026Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization
  6. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  7. 2026Beyond Semantic Manipulation: Token-Space Attacks on Reward Models
  8. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR