mlsanityData to 14 Oct 2026

IBBench-Light: A Paired Evaluation of Task-Conditioned Responses to External Directives

arXiv 2609.13725 ↗First posted 12 Sep 2026cs.AI cs.LOcited by 0

Not on the map. Why?

Abstract (arXiv)

An external record may contain a procedure to apply or text to read, depending on the user's request. IBBench-Light tests both uses against the same record. Twelve semantic bases yield 144 matched pairs per model; four quantized instruction models produced 1,152 archived greedy responses. Paired exact-contract accuracy (PECA) requires both members to satisfy their output contracts. Qwen succeeds on 132 execute and 109 process prompts, but only 97 complete pairs, showing what marginal averages omit. We audit literal-target exposure and case normalization, then add 1,722 logged CPU generations to test directive-absent controls, twelve additional semantic bases, within-base wording changes, and generation stopping. In the pinned Phi rerun, changing the end-of-sequence (EOS) set changes exact paired success from 0/144 to 62/144. A bounded IHEval comparison uses the same SmolLM2 checkpoint and output budget while preserving its published instruction roles and scorer. The benchmark measures conditional task and output-contract success. Its task margins and paired count need to be read together with the stopping policy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 13 of its 15 references that are in the map. First posted on arXiv 12 Sep 2026.

Builds on

13 of 15 references.

References · 13

  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  2. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  3. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  4. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  5. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  6. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  7. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  8. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 125
  9. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  10. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 53
  11. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  12. 2026Defenses Against Prompt Attacks Learn Surface Heuristicscited by 11
1 more
  1. 2026Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defensecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
  2. 2026IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
  3. 2025The Atomic Instruction Gap: Instruction-Tuned LLMs Struggle with Simple, Self-Contained Directives
  4. 2026RIFT: Reordered Instruction Following Testbed To Evaluate Instruction Following in Singular Multistep Prompt Structures
  5. 2025LIFEBench: Evaluating Length Instruction Following in Large Language Models
  6. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
  7. 2026Plan Pointers and Record-Directive Form in Budgeted Verification of Inherited Agent Memory
  8. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions