Alignment has a Fantasia Problem
Not on the map. Why?
Abstract (arXiv)
In accomplishing complex tasks, human cognition typically progresses from abstract to concrete (e.g., from brainstorming ideas to writing an essay). With the advent of highly capable AI assistants, people now offload various parts of their task to the system. However, instruction-tuned AI systems, even when designed to infer implicit intent, lack an understanding of humans' cognitive processes. When a user approaches AI while their goals and intentions are abstract, AI systems often short circuit their cognitive process through which those goals would be refined by jumping toward a final output (e.g., writing the essay entirely). Doing so takes away the user's agency in achieving the task: they may need to spend more time revising or, worse, settle on a suboptimal outcome. We call these failures Fantasia interactions after the famous Disney scene. We argue that Fantasia interactions demand a rethinking of alignment research, where AI systems optimize how cognitive responsibility is allocated within an interaction. We highlight gaps in state-of-the-art alignment methods, and outline a research agenda for training and evaluating models to achieve this vision.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 80 references that are in the map. First posted on arXiv 23 Apr 2026.
Builds on
50 of 80 references.
References · 50
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
- 2022Language Models (Mostly) Know What They Knowcited by 1,856
- 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
- 2023Instruction Tuning for Large Language Models: A Surveycited by 623
- 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
- 2016Cooperative Inverse Reinforcement Learningcited by 300
38 more
- 2023Do Large Language Models Know What They Don't Know?cited by 243
- 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
- 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 154
- 2024Personalized Language Modeling from Personalized Human Feedbackcited by 132
- 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
- 2024The Prompt Report: A Systematic Survey of Prompt Engineering Techniquescited by 121
- 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
- 2025Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Taskcited by 106
- 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
- 2024Personalization of Large Language Models: A Surveycited by 104
- 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
- 2022Evaluating Human-Language Model Interactioncited by 92
- 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
- 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
- 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
- 2024Modeling Future Conversation Turns to Teach LLMs to Ask Clarifying Questionscited by 76
- 2023Bridging the Gulf of Envisioning: Cognitive Design Challenges in LLM Interfacescited by 72
- 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 69
- 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaborationcited by 65
- 2023Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creationcited by 64
- 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 57
- 2023On the steerability of large language models toward data-driven personascited by 53
- 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
- 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
- 2024A Taxonomy for Human-LLM Interaction Modes: An Initial Explorationcited by 38
- 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
- 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
- 2025Language Model Personalization via Reward Factorizationcited by 31
- 2025Navigating Rifts in Human-LLM Grounding: Study and Benchmarkcited by 29
- 2025Generative and Malleable User Interfaces with Generative and Evolving Task-Driven Data Modelcited by 26
- 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
- 2025Active Task Disambiguation with LLMscited by 23
- 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profilescited by 17
- 2024Exploring Prompt Engineering Practices in the Enterprisecited by 11
- 2023Prompt Middleware: Mapping Prompts for Large Language Models to UI Affordancescited by 7
- 2025Feedforward in Generative AI: Opportunities for a Design Spacecited by 5
- 2025User Misconceptions of LLM-Based Conversational Programming Assistantscited by 1
- 2026Creo: From One-Shot Image Generation to Progressive, Co-Creative Ideationcited by 1
Built on by
Not cited yet.