Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
Given the growing influence of language model-based agents on high-stakes societal decisions, from public policy to healthcare, ensuring their beneficial impact requires understanding the far-reaching implications of their suggestions. We propose a proof-of-concept framework that projects how model-generated advice could propagate through societal systems on a macroscopic scale over time, enabling more robust alignment. To assess the long-term safety awareness of language models, we also introduce a dataset of 100 indirect harm scenarios, testing models' ability to foresee adverse, non-obvious outcomes from seemingly harmless user prompts. Our approach achieves not only over 20% improvement on the new dataset but also an average win rate exceeding 70% against strong baselines on existing safety benchmarks (AdvBench, SafeRLHF, WildGuardMix), suggesting a promising direction for safer agents.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 21 of its 39 references that are in the map. First posted on arXiv 26 Jun 2025.
Builds on
21 of 39 references.
References · 21
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
- 2018World Modelscited by 1,444
- 2023Mastering Diverse Domains through World Modelscited by 1,193
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
- 2024Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researcherscited by 348
9 more
- 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
- 2023Large Language Model Alignment: A Surveycited by 241
- 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
- 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 67
- 2023Instruction-tuning Aligns LLMs to the Human Braincited by 39
- 2017Learning model-based planning from scratchcited by 21
- 2023Gradient-based Planning with World Modelscited by 18
- 2023Decoding the Silent Majority: Inducing Belief Augmented Social Graph with Large Language Model for Response Forecastingcited by 11
- 2023Measuring the Effect of Influential Messages on Varying Personascited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Beyond Single-Agent Safety: A Taxonomy of Risks in LLM-to-LLM Interactions
- 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
- 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models
- 2025ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
- 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach
- 2026The Causal Impact of Tool Affordance on Safety Alignment in LLM Agents
- 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment
- 2026Beyond Static Alignment: Hierarchical Policy Control for LLM Safety via Risk-Aware Chain-of-Thought