# Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning

arXiv: https://arxiv.org/abs/2404.10552 · HTML: https://mlsanity.com/paper/2404.10552/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [What Makes Good In-Context Examples for GPT-3?](https://mlsanity.com/paper/2101.06804/) (2021)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations](https://mlsanity.com/paper/2310.06387/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models](https://mlsanity.com/paper/2310.02949/) (2023)
- [Prompt Cache: Modular Attention Reuse for Low-Latency Inference](https://mlsanity.com/paper/2311.04934/) (2023)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability](https://mlsanity.com/paper/2402.08679/) (2024)
- [Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning](https://mlsanity.com/paper/2301.11916/) (2023)
- [On the Relation between Sensitivity and Accuracy in In-context Learning](https://mlsanity.com/paper/2209.07661/) (2022)
- [TIGERScore: Towards Building Explainable Metric for All Text Generation Tasks](https://mlsanity.com/paper/2310.00752/) (2023)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning](https://mlsanity.com/paper/2308.04275/) (2023)

## Built on by (most cited first, 6 of 6)

- [Uncovering Safety Risks of Large Language Models through Concept Activation Vector](https://mlsanity.com/paper/2404.12038/) (2024)
- [AI Safety in Generative AI Large Language Models: A Survey](https://mlsanity.com/paper/2407.18369/) (2024)
- [Revealing Weaknesses in Text Watermarking Through Self-Information Rewrite Attacks](https://mlsanity.com/paper/2505.05190/) (2025)
- [On the (In)Security of LLM App Stores](https://mlsanity.com/paper/2407.08422/) (2024)
- [RLCracker: Evaluating the Worst-Case Vulnerability of LLM Watermarks with Adaptive RL Attacks](https://mlsanity.com/paper/2509.20924/) (2025)
- [LLM Watermark Evasion via Bias Inversion](https://mlsanity.com/paper/2509.23019/) (2025)

## Abstract (arXiv)

The open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. This includes both base models, which are pre-trained on extensive datasets without alignment, and aligned models, deliberately designed to align with ethical standards and human values. Contrary to the prevalent assumption that the inherent instruction-following limitations of base LLMs serve as a safeguard against misuse, our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions. To systematically assess these risks, we introduce a novel set of risk evaluation metrics. Empirical results reveal that the outputs from base LLMs can exhibit risk levels on par with those of models fine-tuned for malicious purposes. This vulnerability, requiring neither specialized knowledge nor training, can be manipulated by almost anyone, highlighting the substantial risk and the critical need for immediate attention to the base LLMs' security protocols.