mlsanityData to 14 Oct 2026

When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems

arXiv 2604.15343 ↗First posted 14 Mar 2026cs.HC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We report a detailed autoethnographic case study of a single-subject who deliberately constructed and operated a multi-modal prompt-engineering system (System A) designed to externalize cognitive self-regulation onto a large language model (LLM). Within 48 hours of the system's completion, a cascade of observable behavioral changes occurred: voluntary transfer of decision-making authority to the LLM, use of LLM-generated output to deflect external criticism, and a loss of self-initiated reasoning that was independently perceived by two uninformed observers, one of whom subsequently became a co-author of this report. We document the precise architectural mechanism responsible: context contamination, whereby prompt-level isolation instructions co-exist with the very emotional and self-referential material they nominally isolate, rendering the isolation directive structurally ineffective within the attention window. We further identify a metacognitive co-option dynamic, in which intact higher-order reasoning capacity was redirected toward defending the closed loop rather than exiting it. Recovery occurred only after physical interruption of the interaction and a self-initiated pharmacologically-mediated sleep event functioning as an external circuit break. A redesigned system (System B) employing physical rather than logical conversation isolation avoided all analogous failure modes. We derive three contributions: (1) a technically-grounded account of why prompt-layer isolation is architecturally insufficient for context-sensitive multi-modal LLM systems; (2) a phenomenological record of closed-loop collapse with external-witness corroboration; and (3) an ethical distinction between protective system design (preventing unintended loss of user agency) and restrictive system design (preventing intentional boundary-pushing), which require fundamentally different account-ability frameworks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 4 of its 33 references that are in the map. First posted on arXiv 14 Mar 2026.

Builds on

4 of 33 references.

References · 4

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  4. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025What Do LLM Agents Do When Left Alone? Evidence of Spontaneous Meta-Cognitive Patterns
  2. 2026Think2: Grounded Metacognitive Reasoning in Large Language Models
  3. 2026LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling
  4. 2025United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory
  5. 2025Emergent Cognitive Convergence via Implementation: Structured Cognitive Loop Reflecting Four Theories of Mind
  6. 2024CogBench: a large language model walks into a psychology lab
  7. 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
  8. 2024Rethinking ChatGPT's Success: Usability and Cognitive Behaviors Enabled by Auto-regressive LLMs' Prompting