mlsanityData to 14 Oct 2026

Step-Level Preference Learning for Generative Agents in Social Simulations

arXiv 2607.14485 ↗First posted 16 Jul 2026cs.AIcited by 1

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

also in 1 more
Abstract (arXiv)

Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 39 references that are in the map. First posted on arXiv 16 Jul 2026.

Builds on

28 of 39 references.

References · 28

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  8. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  10. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  11. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 186
  12. 2023EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activitiescited by 154
16 more
  1. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 111
  2. 2023MetaAgents: Large Language Model Based Agents for Decision-Making on Teamingcited by 85
  3. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 78
  4. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 75
  5. 2023Humanoid Agents: Platform for Simulating Human-like Generative Agentscited by 73
  6. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  7. 2024Simulating Human-like Daily Activities with Desire-driven Autonomycited by 30
  8. 2025CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulationcited by 28
  9. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  10. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Modelscited by 22
  11. 2025Can A Society of Generative Agents Simulate Human Behavior and Inform Public Health Policy? A Case Study on Vaccine Hesitancycited by 17
  12. 2024Evolving Agents: Interactive Simulation of Dynamic and Diverse Human Personalitiescited by 14
  13. 2025Language-Driven Opinion Dynamics in Agent-Based Simulations with LLMscited by 13
  14. 2025YuLan-OneSim: Towards the Next Generation of Social Simulator with Large Language Modelscited by 11
  15. 2025A Visualized Framework for Event Cooperation with Generative Agentscited by 3
  16. 2025SimCity: Multi-Agent Urban Development Simulation with Rich Interactionscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
  2. 2023Generative Agent-Based Modeling: Unveiling Social System Dynamics through Coupling Mechanistic Models with Generative Artificial Intelligence
  3. 2024Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs
  4. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agents
  5. 2023A density estimation perspective on learning from pairwise human preferences
  6. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  7. 2025Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
  8. 2023Human Choice Prediction in Language-based Persuasion Games: Simulation-based Off-Policy Evaluation