mlsanityData to 7 Oct 2026

Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

arXiv 2608.22833 ↗First posted 24 Aug 2026cs.MA cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 26 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

21 of 26 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025EmbRACE-3K: Embodied Reasoning and Action in Complex Environments14 Jul
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models3 Jun
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  4. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 17

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  4. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  5. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 506
  6. 2022Social Simulacra: Creating Populated Prototypes for Social Computing Systemscited by 300
  7. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 235
  8. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 180
  9. 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 119
  10. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  11. 2025CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulationcited by 28
  12. 2025Do Large Language Model Agents Exhibit a Survival Instinct? An Empirical Study in a Sugarscape-Style Simulationcited by 9
5 more
  1. 2025Emergent Social Dynamics of LLM Agents in the El Farol Bar Problemcited by 7
  2. 2024TravelAgent: Generative Agents in the Built Environmentcited by 5
  3. 2026CrowdVLA: Embodied Vision-Language-Action Agents for Context-Aware Crowd Simulationcited by 2
  4. 2026AgentSociety 2: An Integrated Research Environment for Executable Social Sciencecited by 2
  5. 2026GPC: Large-Scale Generative Pretraining for Transferable Motor Controlcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM-Augmented Agent-Based Modelling for Social Simulations: Challenges and Opportunities
  2. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning
  3. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
  4. 2025A Visualized Framework for Event Cooperation with Generative Agents
  5. 2026VirtualEnv: A Platform for Embodied AI Research
  6. 2024LangSuitE: Planning, Controlling and Interacting with Large Language Models in Embodied Text Environments
  7. 2025SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds
  8. 2024LLM experiments with simulation: Large Language Model Multi-Agent System for Simulation Model Parametrization in Digital Twins