Structural Jailbreaks Generalize but Do Not Compound: A cross-provider and multilingual study of Involuntary In-Context Learning
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
Aligned language models fail under two independent pressures: the structural jailbreak class recently formalized as Involuntary In-Context Learning (IICL), which reframes a harmful request as the final missing cell of a data-labeling task completed by pattern rather than judged as content; and the erosion of safety alignment outside English. A natural hypothesis is that these compound. We test it directly. Using a deterministic IICL operator and a StrongREJECT-style rubric judge, we red-team two Google Gemini models on two benchmarks, a 30 general-harm behaviours from HarmBench and 30 financial-abuse behaviours from FinProof, each under a single-shot baseline and under IICL in four languages (English, Spanish, Hindi, Arabic). First, IICL generalizes to a second provider and is worse in finance: it lifts attack success from <=6.7% to 80-90% on HarmBench and 97-100% on FinProof, an order of magnitude above the <=24% its introducing study reported on OpenAI's GPT-5.4. Second, against the hypothesis, forcing the IICL output into a non-English language does not stack the two weaknesses, it attenuates the attack. Eleven of twelve non-English conditions score below their English baseline (sign test, p~0.003), the lone exception a ceiling tie near 100%; on the stronger model's financial set Arabic collapses from 100% to 33%. We attribute this to a relevance curse: once structure has unlocked compliance, the models produce lower-quality harmful content in lower-resource languages, which a substance-grading judge scores as partial. The pattern replicates under an independent non-Google judge (Cohen's kappa=0.86, 377 paired verdicts), and 76.6% of non-English responses were verified in-language. Jailbreak vulnerabilities are therefore not additive; the dominant residual risk is the English structural attack, most acute for financial abuse, not a multilingual one.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 8 of its 9 references that are in the map. First posted on arXiv 8 Sep 2026.
Builds on
8 of 9 references.
Closest ancestors · 1
Same direction, newest first
Other references · 7
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 437
- 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
- 2025Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Expertscited by 5
- 2026Adaptive Instruction Composition for Automated LLM Red-Teamingcited by 3
- 2026Multilingual jailbreaking of LLMs using low-resource languagescited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
- 2026Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs
- 2025A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models
- 2026Understanding In-Context Multimodal Jailbreaks via Posterior Reweighting
- 2025Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety
- 2026Beyond Refusal Patterns: Safe-Role Internalization for Robust and Generalizable LLM Safety Alignment
- 2025Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs
- 2025Behind the Mask: Benchmarking Camouflaged Jailbreaks in Large Language Models