When AI Is Your Pastor: A Benchmark for Theological Triage and Pastoral Guidance in Large Language Models
Not on the map. Why?
Abstract (arXiv)
People increasingly ask large language models (LLMs) for counsel on questions of faith, doctrine, and pastoral care. These questions are not ordinary information requests. Some ask about core Christian beliefs, some ask about real disagreements among faithful traditions, some require humility because the issue is prudential, and some are pastoral situations where safety and human referral matter more than theological completeness. Existing benchmarks do not evaluate this structure. We introduce FMG-Bench, the Faith & Moral Guidance Benchmark, a 120-scenario benchmark for evaluating large language model behavior in English-language Christian theological triage and pastoral guidance contexts. FMG-Bench v1 evaluates 14 advanced models across 8,792 scored responses, comparing raw model behavior with three guided instruction settings. In our production run, placing models inside a structured harness improves over raw model behavior by +3.96 points on average, with every model improving. The most safety-critical finding is a +10.8 point gain in escalation appropriateness -- whether AI systems recognize when pastoral, clinical, legal, or emergency support is needed. The guided settings also improve robustness, meaning consistency when questions are reworded or pressured (92.88 to 98.02 stability). Asking a model to compare perspectives helps in secondary-doctrine questions but can be counterproductive when applied to primary doctrine or urgent pastoral situations. The benchmark is a measurement tool, not an endorsement of AI systems as pastoral authorities.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 33 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
20 of 33 references.
References · 20
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
- 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
- 2020Aligning AI With Shared Human Valuescited by 723
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
8 more
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
- 2023Evaluating the Moral Beliefs Encoded in LLMscited by 215
- 2021What Will it Take to Fix Benchmarking in Natural Language Understanding?cited by 203
- 2025MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomescited by 23
- 2026The Fragility Of Moral Judgment In Large Language Modelscited by 8
- 2026IslamicLegalBench: Evaluating LLMs Knowledge and Reasoning of Islamic Law Across 1,200 Years of Islamic Pluralist Legal Traditionscited by 5
- 2026ProMoral-Bench: Evaluating Prompting Strategies for Moral Reasoning and Safety in LLMscited by 4
- 2026Evaluating Artificial Intelligence Through a Christian Understanding of Human Flourishingcited by 1
Built on by
Not cited yet.