# Mindstorms in Natural Language-Based Societies of Mind

arXiv: https://arxiv.org/abs/2305.17066 · HTML: https://mlsanity.com/paper/2305.17066/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2023.

## Built on by (most cited first, 30 of 107)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [AFlow: Automating Agentic Workflow Generation](https://mlsanity.com/paper/2410.10762/) (2024)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Agent-as-a-Judge: Evaluate Agents with Agents](https://mlsanity.com/paper/2410.10934/) (2024)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://mlsanity.com/paper/2401.12954/) (2024)
- [Secret Collusion among AI Agents: Multi-Agent Deception via Steganography](https://mlsanity.com/paper/2402.07510/) (2024)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Evil Geniuses: Delving into the Safety of LLM-based Agents](https://mlsanity.com/paper/2311.11855/) (2023)
- [Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents](https://mlsanity.com/paper/2402.05746/) (2024)
- [Language Agents as Optimizable Graphs](https://mlsanity.com/paper/2402.16823/) (2024)
- [ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2503.09501/) (2025)
- [FlowReasoner: Reinforcing Query-Level Meta-Agents](https://mlsanity.com/paper/2504.15257/) (2025)
- [ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization](https://mlsanity.com/paper/2502.04306/) (2025)
- [Evaluating Language Model Agency through Negotiations](https://mlsanity.com/paper/2401.04536/) (2024)
- [LLM Voting: Human Choices and AI Collective Decision Making](https://mlsanity.com/paper/2402.01766/) (2024)
- [DetToolChain: A New Prompting Paradigm to Unleash Detection Ability of MLLM](https://mlsanity.com/paper/2403.12488/) (2024)
- [Adaptive In-conversation Team Building for Language Model Agents](https://mlsanity.com/paper/2405.19425/) (2024)
- [Urban Generative Intelligence (UGI): A Foundational Platform for Agents in Embodied City Environment](https://mlsanity.com/paper/2312.11813/) (2023)
- [Multi-Agent Coordination across Diverse Applications: A Survey](https://mlsanity.com/paper/2502.14743/) (2025)
- [AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation](https://mlsanity.com/paper/2312.13010/) (2023)
- [Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models](https://mlsanity.com/paper/2405.16282/) (2024)

## Abstract (arXiv)

Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire diverse societies of large multimodal neural networks (NNs) that solve problems by interviewing each other in a "mindstorm." Recent implementations of NN-based societies of minds consist of large language models (LLMs) and other NN-based experts communicating through a natural language interface. In doing so, they overcome the limitations of single LLMs, improving multimodal zero-shot reasoning. In these natural language-based societies of mind (NLSOMs), new agents -- all communicating through the same universal symbolic language -- are easily added in a modular fashion. To demonstrate the power of NLSOMs, we assemble and experiment with several of them (having up to 129 members), leveraging mindstorms in them to solve some practical AI tasks: visual question answering, image captioning, text-to-image synthesis, 3D generation, egocentric retrieval, embodied AI, and general language-based task solving. We view this as a starting point towards much larger NLSOMs with billions of agents-some of which may be humans. And with this emergence of great societies of heterogeneous minds, many new research questions have suddenly become paramount to the future of artificial intelligence. What should be the social structure of an NLSOM? What would be the (dis)advantages of having a monarchical rather than a democratic structure? How can principles of NN economies be used to maximize the total reward of a reinforcement learning NLSOM? In this work, we identify, discuss, and try to answer some of these questions.