mlsanityData to 7 Oct 2026

Reward Prediction with Factorized World States

arXiv 2603.09400 ↗First posted 10 Mar 2026cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Agents must infer action outcomes and select actions that maximize a reward signal indicating how close the goal is to being reached. Supervised learning of reward models could introduce biases inherent to training data, limiting generalization to novel goals and environments. In this paper, we investigate whether well-defined world state representations alone can enable accurate reward prediction across domains. To address this, we introduce StateFactory, a factorized representation method that transforms unstructured observations into a hierarchical object-attribute structure using language models. This structured representation allows rewards to be estimated naturally as the semantic similarity between the current state and the goal state under hierarchical constraint. Overall, the compact representation structure induced by StateFactory enables strong reward generalization capabilities. We evaluate on RewardPrediction, a new benchmark dataset spanning five diverse domains and comprising 2,454 unique action-observation trajectories with step-wise ground-truth rewards. Our method shows promising zero-shot results against both VLWM-critic and LLM-as-a-Judge reward models, achieving 60% and 8% lower EPIC distance, respectively. Furthermore, this superior reward quality successfully translates into improved agent planning performance, yielding success rate gains of +21.64% on AlfWorld and +12.40% on ScienceWorld over reactive system-1 policies and enhancing system-2 agent planning. Project Page: https://statefactory.github.io

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 82 of its 96 references that are in the map. First posted on arXiv 10 Mar 2026.

Builds on

82 of 96 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Reward Reasoning Model20 May

Other references · 81

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2018World Modelscited by 1,444
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2023Mastering Diverse Domains through World Modelscited by 1,193
  9. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  11. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  12. 2017Hindsight Experience Replaycited by 1,096
  13. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  14. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  15. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  16. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  17. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 556
  18. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  19. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  20. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  21. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  22. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  23. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 372
  24. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  25. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  26. 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
  27. 2022ScienceWorld: Is your Agent Smarter than a 5th Grader?cited by 298
  28. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  29. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  30. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  31. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  32. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  33. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  34. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  35. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  36. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  37. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  38. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Self-Evolving World Models for LLM Agent Planningcited by 5
  2. 2026HyperWorld: Hypergraph-Structured State Serialization Improves Learned Textual World Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling
  2. 2026Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning
  3. 2026Reward as An Agent for Embodied World Models
  4. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models
  5. 2023Task Aware Dreamer for Task Generalization in Reinforcement Learning
  6. 2025Scaling Autonomous Agents via Automatic Reward Modeling And Planning
  7. 2026Causal Reward World Models: Zero-shot Reward Design for Automated Skill Generation
  8. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination