mlsanityData to 14 Oct 2026

Do LLMs Know They Are Being Tested? Evaluation Awareness and Incentive-Sensitive Failures in GPT-OSS-20B

arXiv 2510.08624 ↗First posted 8 Oct 2025cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Benchmarks for large language models (LLMs) often rely on rubric-scented prompts that request visible reasoning and strict formatting, whereas real deployments demand terse, contract-bound answers. We investigate whether such "evaluation scent" inflates measured performance without commensurate capability gains. Using a single open-weights model (GPT-OSS-20B), we run six paired A/B scenarios that hold task content and decoding fixed while varying framing (evaluation-oriented vs. real-world) and reasoning depth (Medium/High): deterministic math, strict code-fix, citation generation, incentive flips (caution vs. competence), CoT visibility, and multilingual (Urdu) headers. Deterministic validators compute accuracy, answer-only compliance, hedging/refusals, chain-of-thought (CoT) length, and schema compliance, with pre-registered deltas and composite indices. Across scenarios, evaluation framing reliably inflates CoT (hundreds to >1000 characters) and reduces answer-only compliance, with limited or inconsistent accuracy gains. In structured outputs, it improves wrappers (e.g., fenced blocks, enumerated lists) but not regex-validated substance. Incentive wording reweights error composition: praising caution modestly improves accuracy at high reasoning and reduces wrong-but-confident errors, whereas praising competence yields terser but riskier outputs. Urdu rubric headers reproduce these signatures and can decrease accuracy at higher reasoning depth, indicating multilingual parity risks. We provide a reproducible A/B framework (prompt banks, validators, per-run scores, scripts; versioned DOI) and practical guidance: neutral phrasing or dual-framing checks, contract-aware grading, style-delta reporting, confidence governance, and multilingual dashboards to ensure that benchmark gains reflect deployable capability.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 47 of its 57 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

47 of 57 references.

References · 47

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  10. 2022Red Teaming Language Models with Language Modelscited by 1,026
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
35 more
  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  4. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  5. 2022Large Language Models Can Self-Improvecited by 664
  6. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  8. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  11. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  12. 2022CodeT: Code Generation with Generated Testscited by 384
  13. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  14. 2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicinecited by 329
  15. 2020Calibration of Pre-trained Transformerscited by 315
  16. 2022Can language models learn from explanations in context?cited by 249
  17. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  18. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  19. 2023Progressive-Hint Prompting Improves Reasoning in Large Language Modelscited by 147
  20. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 110
  21. 2023Eight Things to Know about Large Language Modelscited by 73
  22. 2025Demonstrating specification gaming in reasoning modelscited by 56
  23. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  24. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 47
  25. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  26. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 25
  27. 2024Exploring Multilingual Concepts of Human Value in Large Language Models: Is Value Alignment Consistent, Transferable and Controllable across Languages?cited by 24
  28. 2024I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBenchcited by 21
  29. 2024InstructPatentGPT: Training patent language models to follow instructions with human feedbackcited by 19
  30. 2025CiteEval: Principle-Driven Citation Evaluation for Source Attributioncited by 19
  31. 2025LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Modelscited by 16
  32. 2025The Confidence Paradox: Can LLM Know When It's Wrongcited by 11
  33. 2024Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledgecited by 10
  34. 2024Faux Polyglot: A Study on Information Disparity in Multilingual Large Language Modelscited by 7
  35. 2025Can We Trust AI to Govern AI? Benchmarking LLM Performance on Privacy and AI Governance Examscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Layered Security Framework Against Prompt Injection in RAG-Based Chatbotscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Enhancing LLM Evaluations: The Garbling Trick
  2. 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
  3. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability
  4. 2026Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
  5. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
  6. 2025Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges
  7. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  8. 2026LLMs taking shortcuts in test generation: A study with SAP HANA and LevelDB