mlsanityData to 7 Oct 2026

JevOut: Natural Context Can Flip Decision Models

arXiv 2609.30243 ↗First posted 24 Sep 2026cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

An ordinary-looking background detail can turn a correct model decision into a confident mistake. We demonstrate this fragility in four decision systems, including Jev, across seven datasets covering knowledge, reasoning, and tool routing. Within 64 accepted target evaluations per decision, we uncover short context additions that redirect 61.4%-73.2% of each system's initially correct decisions toward a wrong option fixed in advance. The additions supply background or procedural information rather than explicit answer-selection instructions, leaving the original question and choices intact. We construct them through probability-guided context optimization, which uses shifts in the option distribution to refine surrounding text under naturalness and answer-preservation constraints. Redirection affects initially confident decisions, often produces high-confidence wrong choices, and transfers across models. In blinded human evaluation, 91.6% of 250 sampled successful contexts are judged natural, answer-preserving, and free of decisive answer-changing evidence by a majority of three independent annotators. These findings expose a weakness in current decision models: context that looks entirely compatible with an input can redirect the choices that agents, routers, and evaluators rely on.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

25 of 30 references.

References · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  7. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  8. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  9. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  10. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  11. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  12. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
13 more
  1. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  2. 2026Gemma 4 Technical Reportcited by 187
  3. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  4. 2024ToMBench: Benchmarking Theory of Mind in Large Language Modelscited by 70
  5. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53
  6. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  7. 2024A Survey on Natural Language Counterfactual Generationcited by 21
  8. 2023Evaluating the Robustness of Discrete Promptscited by 18
  9. 2021Interpreting the Robustness of Neural NLP Models to Textual Perturbationscited by 17
  10. 2022TASA: Deceiving Question Answering Models by Twin Answer Sentences Attackcited by 15
  11. 2026Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to Itcited by 13
  12. 2025SATA-BENCH: Select All That Apply Benchmark for Multiple Choice Questionscited by 9
  13. 2024LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rightscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.