mlsanityData to 14 Oct 2026

Pigeonholing: how bad prompts hurt models, causing collapse and mistakes

arXiv 2606.24267 ↗First posted 23 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

While in-context learning is generally shown to be effective in Large Language Models (LLMs), bad contexts can cause performance degradation and mode collapse, a phenomenon we call "pigeonholing." **Unintentionally bad** contexts can happen without malicious jailbreaking intents: For example, a user asks the model to justify an incorrect math theorem or fails to correct the model's buggy code. Specifically, we investigate ``pigeonholing" in two scenarios: (1) when the user suggests a solution, and (2) when the conversation context includes the assistant's previous (incorrect) responses. Our experiments across 10 verifiable and open-ended tasks with 10 different models show that pigeonholing manifests in several ways: (1) repeating the incorrect answers from context (leading to 38-40% performance drop), (2) converging on a narrow set of answers in coding and text generation without exploring alternatives, and (3) flipping stance on controversial topics to align with the user or the assistant's previous claims. We find that pigeonholing worsens almost monotonically with the number of conversation turns (performance drops by additional 14+% as repeated mistakes increase from 1 to 5), and pigeonholing-induced mode collapse can happen even when the provided example is correct. As a step toward mitigation, we propose RLVR with synthetic errors which improves models by 43-60% under bad contexts compared to vanilla RLVR baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 36 of its 59 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

36 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
  2. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack2 Apr
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  11. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
21 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  2. 2025LLMs Get Lost In Multi-Turn Conversationcited by 429
  3. 2023Larger language models do in-context learning differentlycited by 394
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  5. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  6. 2022Can language models learn from explanations in context?cited by 249
  7. 2025Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)cited by 141
  8. 2025ELEPHANT: Measuring and understanding social sycophancy in LLMscited by 129
  9. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  10. 2025Measuring Sycophancy of Language Models in Multi-turn Dialoguescited by 104
  11. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  12. 2024Chain of Attack: a Semantic-Driven Contextual Multi-Turn attacker for LLMcited by 77
  13. 2024From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuningcited by 54
  14. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 40
  15. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
  16. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 31
  17. 2026Sycophantic Chatbots Cause Delusional Spiraling, Even in Ideal Bayesianscited by 21
  18. 2026How LLMs Distort Our Written Languagecited by 21
  19. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMscited by 14
  20. 2025A Simple "Try Again" Can Elicit Multi-Turn LLM Reasoningcited by 11
  21. 2026Improving Interactive In-Context Learning from Natural Language Feedbackcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Response Attack: Exploiting Contextual Priming to Jailbreak Large Language Models
  2. 2025Learning to Extract Context for Context-Aware LLM Inference
  3. 2024The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance
  4. 2025Exploring the Secondary Risks of Large Language Models
  5. 2026Contextual Drag: How Errors in the Context Affect LLM Reasoning
  6. 2026The Illusion of Robustness: Aggregate Accuracy Hides Prediction Flips under Task-Irrelevant Context
  7. 2025No Need for Explanations: LLMs can implicitly learn from mistakes in-context
  8. 2025Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting