# Minimal Local Simulation Foundations for LLM- and VLM-Driven Agents in 2D and 3D Environments

arXiv: https://arxiv.org/abs/2608.22833 · HTML: https://mlsanity.com/paper/2608.22833/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 21 of 21)

- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Society](https://mlsanity.com/paper/2502.08691/) (2025)
- [Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordia](https://mlsanity.com/paper/2312.03664/) (2023)
- [OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2506.03135/) (2025)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [CitySim: Modeling Urban Behaviors and City Dynamics with Large-Scale LLM-Driven Agent Simulation](https://mlsanity.com/paper/2506.21805/) (2025)
- [Do Large Language Model Agents Exhibit a Survival Instinct? An Empirical Study in a Sugarscape-Style Simulation](https://mlsanity.com/paper/2508.12920/) (2025)
- [Emergent Social Dynamics of LLM Agents in the El Farol Bar Problem](https://mlsanity.com/paper/2509.04537/) (2025)
- [TravelAgent: Generative Agents in the Built Environment](https://mlsanity.com/paper/2412.18985/) (2024)
- [EmbRACE-3K: Embodied Reasoning and Action in Complex Environments](https://mlsanity.com/paper/2507.10548/) (2025)
- [CrowdVLA: Embodied Vision-Language-Action Agents for Context-Aware Crowd Simulation](https://mlsanity.com/paper/2604.05525/) (2026)
- [GPC: Large-Scale Generative Pretraining for Transferable Motor Control](https://mlsanity.com/paper/2606.29148/) (2026)
- [AgentSociety 2: An Integrated Research Environment for Executable Social Science](https://mlsanity.com/paper/2607.11895/) (2026)

## Abstract (arXiv)

Large language models (LLMs) and vision-language models (VLMs) are expanding the range of behaviors that can be represented in agent-based simulations, but many contemporary platforms are difficult to study, modify, or run on ordinary computers. We present two intentionally minimal simulation foundations for education and rapid prototyping. SD-AgentFoundry-2D provides a two-dimensional multi-agent environment in which locally hosted LLM agents move, communicate, respond to place occupancy, and encounter spatially localized fire events. SD-AgentFoundry-3D provides a three-dimensional digital-twin environment in which a locally hosted VLM receives first-person images and produces natural-language movement instructions. Both codebases are designed to run locally on macOS, Windows, and Linux and are deliberately left open to modification rather than developed as finished applications. Together, they offer accessible starting points for learning about generative social simulation and for building domain-specific extensions.