# RAIL in the Wild: Operationalizing Responsible AI Evaluation Using Anthropic's Value Dataset

arXiv: https://arxiv.org/abs/2505.00204 · HTML: https://mlsanity.com/paper/2505.00204/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2025.

## Built on by (most cited first, 3 of 3)

- [Autonomy by Design: Preserving Human Autonomy in AI Decision-Support](https://mlsanity.com/paper/2506.23952/) (2025)
- [When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift](https://mlsanity.com/paper/2605.25629/) (2026)
- [RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for LLM Agents](https://mlsanity.com/paper/2607.16215/) (2026)

## Abstract (arXiv)

As AI systems become embedded in real-world applications, ensuring they meet ethical standards is crucial. While existing AI ethics frameworks emphasize fairness, transparency, and accountability, they often lack actionable evaluation methods. This paper introduces a systematic approach using the Responsible AI Labs (RAIL) framework, which includes eight measurable dimensions to assess the normative behavior of large language models (LLMs). We apply this framework to Anthropic's "Values in the Wild" dataset, containing over 308,000 anonymized conversations with Claude and more than 3,000 annotated value expressions. Our study maps these values to RAIL dimensions, computes synthetic scores, and provides insights into the ethical behavior of LLMs in real-world use.