# Blind Judgement: Agent-Based Supreme Court Modelling With GPT

arXiv: https://arxiv.org/abs/2301.05327 · HTML: https://mlsanity.com/paper/2301.05327/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 28 references that are in the map.
- First posted on arXiv 12 Jan 2023.

## Placed in

- LLM agents and tool use / [Social simulation and multi-agent debate](https://mlsanity.com/agents/social-simulation-and-multi-agent-debate/)

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level](https://mlsanity.com/paper/2112.15594/) (2021)
- [Generalizing over Long Tail Concepts for Medical Term Normalization](https://mlsanity.com/paper/2210.11947/) (2022)
- [Natural-Language Multi-Agent Simulations of Argumentative Opinion Dynamics](https://mlsanity.com/paper/2104.06737/) (2021)
- [The COVID That Wasn't: Counterfactual Journalism Using GPT](https://mlsanity.com/paper/2210.06644/) (2022)

## Built on by (most cited first, 30 of 38)

- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Challenges and Applications of Large Language Models](https://mlsanity.com/paper/2307.10169/) (2023)
- [CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations](https://mlsanity.com/paper/2310.11501/) (2023)
- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Explaining Legal Concepts with Augmented Large Language Models (GPT-4)](https://mlsanity.com/paper/2306.09525/) (2023)
- [AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents](https://mlsanity.com/paper/2408.08089/) (2024)
- [Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents](https://mlsanity.com/paper/2405.10467/) (2024)
- [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [Simulating Human Strategic Behavior: Comparing Single and Multi-agent LLMs](https://mlsanity.com/paper/2402.08189/) (2024)
- [Generative Agent-Based Modeling: Unveiling Social System Dynamics through Coupling Mechanistic Models with Generative Artificial Intelligence](https://mlsanity.com/paper/2309.11456/) (2023)
- [LLM-based multi-agent poetry generation in non-cooperative environments](https://mlsanity.com/paper/2409.03659/) (2024)
- [Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agents](https://mlsanity.com/paper/2311.13148/) (2023)
- [An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making](https://mlsanity.com/paper/2410.15168/) (2024)
- [DEEM: Dynamic Experienced Expert Modeling for Stance Detection](https://mlsanity.com/paper/2402.15264/) (2024)
- [Smart Agent-Based Modeling: On the Use of Large Language Models in Computer Simulations](https://mlsanity.com/paper/2311.06330/) (2023)
- [BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks](https://mlsanity.com/paper/2410.12974/) (2024)
- [Agents on the Bench: Large Language Model Based Multi Agent Framework for Trustworthy Digital Justice](https://mlsanity.com/paper/2412.18697/) (2024)
- [Algorithmic Ghost in the Research Shell: Large Language Models and Academic Knowledge Creation in Management Research](https://mlsanity.com/paper/2303.07304/) (2023)
- [LLMediator: GPT-4 Assisted Online Dispute Resolution](https://mlsanity.com/paper/2307.16732/) (2023)
- [It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human](https://mlsanity.com/paper/2407.06798/) (2024)
- [Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice](https://mlsanity.com/paper/2412.15260/) (2024)
- [SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in India](https://mlsanity.com/paper/2509.03793/) (2025)
- [Synthetic Social Media Influence Experimentation via an Agentic Reinforcement Learning Large Language Model Bot](https://mlsanity.com/paper/2411.19635/) (2024)
- [Large Language Model-Based Agents for Automated Research Reproducibility: An Exploratory Study in Alzheimer's Disease](https://mlsanity.com/paper/2505.23852/) (2025)

## Abstract (arXiv)

We present a novel Transformer-based multi-agent system for simulating the judicial rulings of the 2010-2016 Supreme Court of the United States. We train nine separate models with the respective authored opinions of each supreme justice active ca. 2015 and test the resulting system on 96 real-world cases. We find our system predicts the decisions of the real-world Supreme Court with better-than-random accuracy. We further find a correlation between model accuracy with respect to individual justices and their alignment between legal conservatism & liberalism. Our methods and results hold significance for researchers interested in using language models to simulate politically-charged discourse between multiple agents.