mlsanityData to 14 Oct 2026

Evaluating the Goal-Directedness of Large Language Models

arXiv 2504.11844 ↗First posted 16 Apr 2025cs.AI cs.CL cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

To what extent do LLMs use their capabilities towards their given goal? We take this as a measure of their goal-directedness. We evaluate goal-directedness on tasks that require information gathering, cognitive effort, and plan execution, where we use subtasks to infer each model's relevant capabilities. Our evaluations of LLMs from Google DeepMind, OpenAI, and Anthropic show that goal-directedness is relatively consistent across tasks, differs from task performance, and is only moderately sensitive to motivational prompts. Notably, most models are not fully goal-directed. We hope our goal-directedness evaluations will enable better monitoring of LLM progress, and enable more deliberate design choices of agentic properties in LLMs.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 36 of its 66 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

36 of 66 references.

References · 36

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  7. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  8. 2024Understanding the planning of LLM agents: A surveycited by 421
  9. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  10. 2024Alignment faking in large language modelscited by 360
  11. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  12. 2024Large Language Models for Mathematical Reasoning: Progresses and Challengescited by 284
24 more
  1. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 223
  2. 2024AI Agents That Mattercited by 219
  3. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 179
  4. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  5. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  6. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  7. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  8. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  9. 2024The Ethics of Advanced AI Assistantscited by 107
  10. 2022Is Power-Seeking AI an Existential Risk?cited by 105
  11. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 94
  12. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  13. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  14. 2024Planning Anything with Rigor: General-Purpose Zero-Shot Planning with LLM-based Formalized Programmingcited by 35
  15. 2022Discovering Agentscited by 28
  16. 2022Context-Aware Language Modeling for Goal-Oriented Dialogue Systemscited by 22
  17. 2024Are Long-LLMs A Necessity For Long-Context Tasks?cited by 22
  18. 2024SelfGoal: Your Language Agents Already Know How to Achieve High-level Goalscited by 21
  19. 2022Path-Specific Objectives for Safer Agent Incentivescited by 20
  20. 2018Agents and Devices: A Relative Definition of Agencycited by 16
  21. 2024Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Modelscited by 16
  22. 2023Can LLMs Fix Issues with Reasoning Models? Towards More Likely Models for AI Planningcited by 8
  23. 2024Towards Measuring Goal-Directedness in AI Systemscited by 7
  24. 2024Measuring Goal-Directednesscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  2. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  3. 2026SafePred: A Predictive Guardrail for Computer-Using Agents via World Modelscited by 12
  4. 2025Towards Strategic Persuasion with Language Modelscited by 10
  5. 2025What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignmentcited by 8
  6. 2026Propensity Inference: Environmental Contributors to LLM Behaviourcited by 7
  7. 2026Training Agents to Self-Report Misbehaviorcited by 6
  8. 2025Goal-Directedness is in the Eye of the Beholdercited by 3
  9. 2025Technical Report: Evaluating Goal Drift in Language Model Agentscited by 2
  10. 2025BioBlue: Systematic runaway-optimiser-like LLM failure modes on biologically and economically aligned AI safety benchmarks for LLMs with simplified observation formatcited by 1
  11. 2026A Behavioural and Representational Evaluation of Goal-Directedness in Language Model Agentscited by 1
  12. 2026Motivation in Large Language Modelscited by 1
4 more
  1. 2026Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Frameworkcited by 1
  2. 2026Causal Foundations of Collective Agencycited by 0
  3. 2026SchemeArena: Factorized Stress Testing of Scheming in LLM Agentscited by 0
  4. 2026Shutdown Sabotage Propensities in Multi-Agent Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Language Model Goal Selection Differs from Humans' in a Self-Directed Learning Task
  2. 2025Do Large Language Models Know What They Are Capable Of?
  3. 2026Agentic Cognitive Depth: Operational Criteria for Evaluating LLM Agents
  4. 2025The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution
  5. 2025On the Limits of Innate Planning in Large Language Models
  6. 2023Establishing Trustworthiness: Rethinking Tasks and Model Evaluation
  7. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
  8. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents