# LongSafety: Evaluating Long-Context Safety of Large Language Models

arXiv: https://arxiv.org/abs/2502.16971 · HTML: https://mlsanity.com/paper/2502.16971/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 24 references that are in the map.
- First posted on arXiv 24 Feb 2025.

## Builds on (most cited first, 13 of 13)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models](https://mlsanity.com/paper/2402.05044/) (2024)
- [Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization](https://mlsanity.com/paper/2311.09096/) (2023)
- [Cognitive Overload Attack:Prompt Injection for Long Context](https://mlsanity.com/paper/2410.11272/) (2024)
- [LongSafety: Enhance Safety for Long-Context LLMs](https://mlsanity.com/paper/2411.06899/) (2024)

## Built on by (most cited first, 12 of 12)

- [ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMs](https://mlsanity.com/paper/2505.14035/) (2025)
- [The Side Effects of Being Smart: Safety Risks in MLLMs' Multi-Image Reasoning](https://mlsanity.com/paper/2601.14127/) (2026)
- [Jailbreaking in the Haystack](https://mlsanity.com/paper/2511.04707/) (2025)
- [HauntAttack: When Attack Follows Reasoning as a Shadow](https://mlsanity.com/paper/2506.07031/) (2025)
- [Beyond Context: Large Language Models' Failure to Grasp Users' Intent](https://mlsanity.com/paper/2512.21110/) (2025)
- [Learning of Robot Safety Policies via Adversarial Synthetic Scenarios](https://mlsanity.com/paper/2606.05952/) (2026)
- [FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming](https://mlsanity.com/paper/2606.19887/) (2026)
- [Long-Term Simulation Exposes Cognitive-Developmental Risks in AI Companions](https://mlsanity.com/paper/2606.25396/) (2026)
- [Structured Semantic Cloaking for Jailbreak Attacks on Large Language Models](https://mlsanity.com/paper/2603.16192/) (2026)
- [Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models](https://mlsanity.com/paper/2607.02914/) (2026)
- [When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systems](https://mlsanity.com/paper/2609.23700/) (2026)
- [DistressBench: Evaluating Multi-Dimensional Crisis Support in Large Language Models](https://mlsanity.com/paper/2610.12758/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) continue to advance in understanding and generating long sequences, new safety concerns have been introduced through the long context. However, the safety of LLMs in long-context tasks remains under-explored, leaving a significant gap in both evaluation and improvement of their safety. To address this, we introduce LongSafety, the first comprehensive benchmark specifically designed to evaluate LLM safety in open-ended long-context tasks. LongSafety encompasses 7 categories of safety issues and 6 user-oriented long-context tasks, with a total of 1,543 test cases, averaging 5,424 words per context. Our evaluation towards 16 representative LLMs reveals significant safety vulnerabilities, with most models achieving safety rates below 55%. Our findings also indicate that strong safety performance in short-context scenarios does not necessarily correlate with safety in long-context tasks, emphasizing the unique challenges and urgency of improving long-context safety. Moreover, through extensive analysis, we identify challenging safety issues and task types for long-context models. Furthermore, we find that relevant context and extended input sequences can exacerbate safety risks in long-context scenarios, highlighting the critical need for ongoing attention to long-context safety challenges. Our code and data are available at https://github.com/thu-coai/LongSafety.