mlsanityData to 7 Oct 2026

Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs

arXiv 2601.21233 ↗First posted 29 Jan 2026cs.AIcited by 3

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Autonomous code agents built on large language models are reshaping software and AI development through tool use, long-horizon reasoning, and self-directed interaction. However, this autonomy introduces a previously unrecognized security risk: agentic interaction fundamentally expands the LLM attack surface, enabling systematic probing and recovery of hidden system prompts that guide model behavior. We identify system prompt extraction as an emergent vulnerability intrinsic to code agents and present textbftextscJustAsk, a self-evolving framework that autonomously discovers effective extraction strategies through interaction alone. Unlike prior prompt-engineering or dataset-based attacks, textscJustAsk requires no handcrafted prompts, labeled supervision, or privileged access beyond standard user interaction. It formulates extraction as an online exploration problem, using Upper Confidence Bound-based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration. These skills exploit imperfect system-instruction generalization and inherent tensions between helpfulness and safety. Evaluated on 41 black-box commercial models across multiple providers, textscJustAsk consistently achieves full or near-complete system prompt recovery, revealing recurring design- and architecture-level vulnerabilities. Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 32 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

22 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety2 Feb
  2. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage7 Dec

Other references · 20

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  6. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  7. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  8. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  9. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  10. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  11. 2024Foundation Policies with Hilbert Representationscited by 89
  12. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
8 more
  1. 2023Effective Prompt Extraction from Language Modelscited by 83
  2. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  3. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 61
  4. 2024Extracting Prompts by Inverting LLM Outputscited by 40
  5. 2024Prompt Stealing Attacks Against Large Language Modelscited by 39
  6. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychologycited by 25
  7. 2024Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applicationscited by 16
  8. 2025ProxyPrompt: Securing System Prompts against Prompt Extraction Attackscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Safety in Self-Evolving LLM Agent Systems: Threats, Amplification, and Case Studiescited by 6
  2. 2026Concepts for Securing Agentic AI Coding and the Terok Environmentcited by 0
  3. 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Mapping the Exploitation Surface: A 10,000-Trial Taxonomy of What Makes LLM Agents Exploit Vulnerabilities
  2. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  3. 2026MulVul: Retrieval-augmented Multi-Agent Code Vulnerability Detection via Cross-Model Prompt Evolution
  4. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems
  5. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomy
  6. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
  7. 2026Arbiter: Detecting Interference in LLM Agent System Prompts
  8. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation