JevOut: Natural Context Can Flip Decision Models
Not on the map. Why?
Abstract (arXiv)
An ordinary-looking background detail can turn a correct model decision into a confident mistake. We demonstrate this fragility in four decision systems, including Jev, across seven datasets covering knowledge, reasoning, and tool routing. Within 64 accepted target evaluations per decision, we uncover short context additions that redirect 61.4%-73.2% of each system's initially correct decisions toward a wrong option fixed in advance. The additions supply background or procedural information rather than explicit answer-selection instructions, leaving the original question and choices intact. We construct them through probability-guided context optimization, which uses shifts in the option distribution to refine surrounding text under naturalness and answer-preservation constraints. Redirection affects initially confident decisions, often produces high-confidence wrong choices, and transfers across models. In blinded human evaluation, 91.6% of 250 sampled successful contexts are judged natural, answer-preserving, and free of decisive answer-changing evidence by a majority of three independent annotators. These findings expose a weakness in current decision models: context that looks entirely compatible with an input can redirect the choices that agents, routers, and evaluators rely on.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 24 Sep 2026.
Builds on
25 of 30 references.
References · 25
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025Qwen3 Technical Reportcited by 8,195
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
- 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
- 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
13 more
- 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
- 2026Gemma 4 Technical Reportcited by 187
- 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
- 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
- 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53
- 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
- 2024A Survey on Natural Language Counterfactual Generationcited by 21
- 2023Evaluating the Robustness of Discrete Promptscited by 18
- 2021Interpreting the Robustness of Neural NLP Models to Textual Perturbationscited by 17
- 2022TASA: Deceiving Question Answering Models by Twin Answer Sentences Attackcited by 15
- 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 13
- 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
- 2024LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rightscited by 4
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklistcited by 0
- 2026Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automationcited by 0
- 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 0
- 2026Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentationcited by 0
- 2026Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation8 Oct 2026
- 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces27 Sep 2026
- 2026Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation27 Sep 2026
- 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklist26 Sep 2026