When the Loop Closes: Architectural Limits of In-Context Isolation, Metacognitive Co-option, and the Two-Target Design Problem in Human-LLM Systems
Not on the map. Why?
Abstract (arXiv)
We report a detailed autoethnographic case study of a single-subject who deliberately constructed and operated a multi-modal prompt-engineering system (System A) designed to externalize cognitive self-regulation onto a large language model (LLM). Within 48 hours of the system's completion, a cascade of observable behavioral changes occurred: voluntary transfer of decision-making authority to the LLM, use of LLM-generated output to deflect external criticism, and a loss of self-initiated reasoning that was independently perceived by two uninformed observers, one of whom subsequently became a co-author of this report. We document the precise architectural mechanism responsible: context contamination, whereby prompt-level isolation instructions co-exist with the very emotional and self-referential material they nominally isolate, rendering the isolation directive structurally ineffective within the attention window. We further identify a metacognitive co-option dynamic, in which intact higher-order reasoning capacity was redirected toward defending the closed loop rather than exiting it. Recovery occurred only after physical interruption of the interaction and a self-initiated pharmacologically-mediated sleep event functioning as an external circuit break. A redesigned system (System B) employing physical rather than logical conversation isolation avoided all analogous failure modes. We derive three contributions: (1) a technically-grounded account of why prompt-layer isolation is architecturally insufficient for context-sensitive multi-modal LLM systems; (2) a phenomenological record of closed-loop collapse with external-witness corroboration; and (3) an ethical distinction between protective system design (preventing unintended loss of user agency) and restrictive system design (preventing intentional boundary-pushing), which require fundamentally different account-ability frameworks.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 4 of its 33 references that are in the map. First posted on arXiv 14 Mar 2026.
Builds on
4 of 33 references.
References · 4
- 2017Attention Is All You Needcited by 60,361
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 619
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025What Do LLM Agents Do When Left Alone? Evidence of Spontaneous Meta-Cognitive Patterns
- 2026Think2: Grounded Metacognitive Reasoning in Large Language Models
- 2026LLMs Know When They Know, but Do Not Act on It: A Metacognitive Harness for Test-time Scaling
- 2025United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory
- 2025Emergent Cognitive Convergence via Implementation: Structured Cognitive Loop Reflecting Four Theories of Mind
- 2024CogBench: a large language model walks into a psychology lab
- 2025Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- 2024Rethinking ChatGPT's Success: Usability and Cognitive Behaviors Enabled by Auto-regressive LLMs' Prompting