mlsanityData to 14 Oct 2026

Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling

arXiv 2407.02446 ↗First posted 2 Jul 2024cs.CL cs.AIcited by 20

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation, yet they struggle with one foundational task: next-token prediction. As RLHF models become agent models aimed at interacting with humans, they seem to lose their world modeling -- the ability to predict what comes next in arbitrary documents, which is the foundational training objective of the Base LMs that RLHF adapts. Besides empirically demonstrating this trade-off, we propose a potential explanation: to perform coherent long-form generation, RLHF models restrict randomness via implicit blueprints. In particular, RLHF models concentrate probability on sets of anchor spans that co-occur across multiple generations for the same prompt, serving as textual scaffolding but also limiting a model's ability to generate documents that do not include these spans. We study this trade-off on the most effective current agent models, those aligned with RLHF, while exploring why this may remain a fundamental trade-off between models that act and those that predict, even as alignment techniques improve.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 29 of its 36 references that are in the map. First posted on arXiv 2 Jul 2024.

Builds on

29 of 36 references.

References · 29

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2023LIMA: Less Is More for Alignmentcited by 1,100
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  11. 2018Which Training Methods for GANs do actually Converge?cited by 679
  12. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
17 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  2. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 503
  3. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  4. 2023How Language Model Hallucinations Can Snowballcited by 311
  5. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 297
  6. 2023Self-Consuming Generative Models Go MADcited by 250
  7. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  8. 2016Mode Regularized Generative Adversarial Networkscited by 123
  9. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  10. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  11. 2019On the Use of ArXiv as a Datasetcited by 103
  12. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  13. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 79
  14. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  15. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  16. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignmentcited by 33
  17. 2019Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehensioncited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  2. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 77
  3. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 51
  4. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 39
  5. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  6. 2025Improved Representation Steering for Language Modelscited by 21
  7. 2024Extracting Paragraphs from LLM Token Activationscited by 9
  8. 2024Diversity-Rewarded CFG Distillationcited by 9
  9. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generationcited by 8
  10. 2026Beyond Next-Observation Prediction: Agent-Authored World Modeling for Sequential Decision Makingcited by 8
  11. 2024Towards Safe and Honest AI Agents with Neural Self-Other Overlapcited by 6
  12. 2025Poor Alignment and Steerability of Large Language Models: Evidence from College Admission Essayscited by 6
8 more
  1. 2025BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Modelscited by 5
  2. 2025Conditioning LLMs to Generate Code-Switched Textcited by 3
  3. 2025Generation Space Size: Understanding and Calibrating Open-Endedness of LLM Generationscited by 2
  4. 2025ParaScopes: What do Language Models Activations Encode About Future Text?cited by 2
  5. 2026AI Coding Agents in Social Science: Methodologically Diverse, Empirically Consistent, Interpretively Vulnerablecited by 1
  6. 2026Safety from Honesty in a Disinterested AI Predictorcited by 1
  7. 2026Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Modelscited by 1
  8. 2025Sample, Align, Synthesize: Graph-Based Response Synthesis with ConGrscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts
  2. 2024Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
  3. 2026Bridging the Agent-World Gap: Text World Models for LLM-based Agents
  4. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agents
  5. 2026From Tokens To Agents: A Researcher's Guide To Understanding Large Language Models
  6. 2026HyperWorld: Hypergraph-Structured State Serialization Improves Learned Textual World Models
  7. 2026The Illusion of Stochasticity in LLMs
  8. 2025From Word to World: Can Large Language Models be Implicit Text-based World Models?