mlsanityData to 14 Oct 2026

A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving

arXiv 2605.27763 ↗First posted 26 May 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Safety evaluations of language models often treat serving configuration as fixed background infrastructure, but batch condition is an untested treatment variable whenever the same prompt may be evaluated alone, in a synchronized batch, or inside a continuous-batching scheduler. We synthesize four artifact-backed studies into a paired testing protocol: Study A combines local discovery, scorer-corrected adjudication, and true-batching confirmation; Study B tests cross-model generalization; Study C tests continuous-batch composition; and Study D runs a batch-invariant-kernel ablation. The local test finds safety-label changes more often than capability-label changes (0.51% vs. 0.14%), but adjudication of 63 candidate rows leaves only 17 genuine behavioral flips, implying a corrected full-set rate of 0.16%. The 15-model extension finds no detectable universal safety-over-capability skew: flips are near parity (0.94x), alignment type has no detectable association (p=0.942, η2=0.033), and output instability is the strongest tested fragility screen (r=0.909, bootstrap 95% CI [0.65, 0.97]). In the targeted kernel ablation, standard vLLM reproduces 22/55 label flips on current score-flip candidates, while enabling VLLMBATCHINVARIANT=1 reduces the same test to 0/55 flips; the composition test separately finds no aggregate effect at 4.7pp sensitivity. The testing recommendation is exact-stack validation: evaluate refusal at the served batch setting, pair safety prompts with capability controls, and report low-rate directional flips separately from aggregate null effects.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 30 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

21 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar

Other references · 20

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  11. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  12. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
8 more
  1. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  4. 2024Non-Determinism of "Deterministic" LLM Settingscited by 92
  5. 2024Exploiting LLM Quantizationcited by 65
  6. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 57
  7. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 16
  8. 2026Q-realign: Piggybacking Realignment on Quantization for Safe and Efficient LLM Deploymentcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting
  2. 2025The Instability of Safety: How Random Seeds and Temperature Expose Inconsistent LLM Refusal Behavior
  3. 2026Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing
  4. 2026Measuring Evaluation-Context Divergence in Open-Weight LLMs: A Paired-Prompt Protocol with Pilot Evidence of Alignment-Pipeline-Specific Heterogeneity
  5. 2025When Refusals Fail: Unstable Safety Mechanisms in Long-Context LLM Agents
  6. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMs
  7. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  8. 2026When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs