mlsanityData to 14 Oct 2026

Adaptively profiling models with task elicitation

arXiv 2503.01986 ↗First posted 3 Mar 2025cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Language model evaluations often fail to characterize consequential failure modes, forcing experts to inspect outputs and build new benchmarks. We introduce task elicitation, a method that automatically builds new evaluations to profile model behavior. Task elicitation finds hundreds of natural-language tasks -- an order of magnitude more than prior work -- where frontier models exhibit systematic failures, in domains ranging from forecasting to online harassment. For example, we find that Sonnet 3.5 over-associates quantum computing and AGI and that o3-mini is prone to hallucination when fabrications are repeated in-context.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

35 of 71 references.

References · 35

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  11. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  12. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
23 more
  1. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  2. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 163
  3. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  4. 2024Evaluating Frontier Models for Dangerous Capabilitiescited by 128
  5. 2024Approaching Human-Level Forecasting with Language Modelscited by 107
  6. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  7. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 98
  8. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 75
  9. 2023Skill-Mix: a Flexible and Expandable Family of Evaluations for AI modelscited by 64
  10. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  11. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 46
  12. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 41
  13. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 40
  14. 2024Consistency Checks for Language Model Forecasterscited by 27
  15. 2023From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Modelscited by 26
  16. 2025Eliciting Language Model Behaviors with Investigator Agentscited by 23
  17. 2025Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboardscited by 20
  18. 2024Evaluating Large Language Models' Capability to Launch Fully Automated Spear Phishing Campaigns: Validated on Human Subjectscited by 17
  19. 2024See What LLMs Cannot Answer: A Self-Challenge Framework for Uncovering LLM Weaknessescited by 15
  20. 2024Building Knowledge-Guided Lexica to Model Cultural Variationcited by 11
  21. 2024Report Cards: Qualitative Evaluation of Language Models Using Natural Language Summariescited by 6
  22. 2024Logicbreaks: A Framework for Understanding Subversion of Rule-based Inferencecited by 4
  23. 2025Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstructioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025BrowserArena: Evaluating LLM Agents on Real-World Web Navigation Taskscited by 14
  2. 2026HarnessEval-W: Agentifying the Evaluation of Visual Worldscited by 4
  3. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Modelscited by 2
  4. 2026BLOOM-WILT: Logit Tilting for Behaviour Elicitation in Automated LLM Auditingcited by 0