mlsanityData to 7 Oct 2026

Forecasting Rare Language Model Behaviors

arXiv 2502.16797 ↗First posted 24 Feb 2025cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Standard language model evaluations can fail to capture risks that emerge only at deployment scale. For example, a model may produce safe responses during a small-scale beta test, yet reveal dangerous information when processing billions of requests at deployment. To remedy this, we introduce a method to forecast potential risks across orders of magnitude more queries than we test during evaluation. We make forecasts by studying each query's elicitation probability -- the probability the query produces a target behavior -- and demonstrate that the largest observed elicitation probabilities predictably scale with the number of queries. We find that our forecasts can predict the emergence of diverse undesirable behaviors -- such as assisting users with dangerous chemical synthesis or taking power-seeking actions -- across up to three orders of magnitude of query volume. Our work enables model developers to proactively anticipate and patch rare failures before they manifest during large-scale deployments.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 33 of its 51 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

33 of 51 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  8. 2022Emergent Abilities of Large Language Modelscited by 2,483
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
21 more
  1. 2018Model Cards for Model Reportingcited by 1,068
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  4. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  7. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  8. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  11. 2023LeanDojo: Theorem Proving with Retrieval-Augmented Language Modelscited by 414
  12. 2024A StrongREJECT for Empty Jailbreakscited by 352
  13. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  14. 2020Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claimscited by 163
  15. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  16. 2021Keeping it Simple: Language Models can learn Complex Molecular Distributionscited by 68
  17. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 60
  18. 2018A Statistical Approach to Assessing Neural Network Robustnesscited by 54
  19. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  20. 2024Adaptive Deployment of Untrusted LLMs Reduces Distributed Threatscited by 27
  21. 2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 12.

16 papers.

Sort:

By area:

  1. 2025Efficient Prediction of Pass@k Scaling in Large Language Modelscited by 14
  2. 2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
  3. 2026Rare Event Analysis of Large Language Modelscited by 6
  4. 2026Estimating Tail Risks in Language Model Output Distributionscited by 6
  5. 2026Abstractive Red-Teaming of Language Model Charactercited by 1
  6. 2026Comprehensive AI governance requires addressing non-model gainscited by 1
  7. 2026SCARCE: Scalable Cascade Analysis for Rare-event Characterisation via Embeddingscited by 1
  8. 2026Where do LLMs Fall Short in CBT-Guided Affective Reasoning?cited by 1
  9. 2026Estimating Rare Events in Language Models with Proper Evaluationcited by 1
  10. 2025`A la recherche du sens perdu: your favourite LLM might have more to say than you can understandcited by 0
  11. 2025The Law-Following AI Framework: Legal Foundations and Technical Constraints. Legal Analogues for AI Actorship and technical feasibility of Law Alignmentcited by 0
  12. 2026Forecasting Future Behavior as a Learning Taskcited by 0
4 more
  1. 2026Sound Probabilistic Safety Bounds for Large Language Modelscited by 0
  2. 2026Contract monitoring: governing AI via separation of powerscited by 0
  3. 2026Quantifying Behavioral Tails in Black-Box Language Modelscited by 0
  4. 2026Estimating great expectations under autoregressive language models with potentialscited by 0