# NodeSynth: Socially Aligned Synthetic Data for AI Evaluation

arXiv: https://arxiv.org/abs/2605.14381 · HTML: https://mlsanity.com/paper/2605.14381/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 24 of 24)

- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications](https://mlsanity.com/paper/2311.08592/) (2023)
- [Learning diverse attacks on large language models for robust red-teaming and safety tuning](https://mlsanity.com/paper/2405.18540/) (2024)
- [A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models](https://mlsanity.com/paper/2403.12025/) (2024)
- [Evaluating Language Models as Synthetic Data Generators](https://mlsanity.com/paper/2412.03679/) (2024)
- [Efficacy of Synthetic Data as a Benchmark](https://mlsanity.com/paper/2409.11968/) (2024)
- [Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction](https://mlsanity.com/paper/2409.16783/) (2024)
- [Synthetic Data in AI: Challenges, Applications, and Ethical Implications](https://mlsanity.com/paper/2401.01629/) (2024)
- [Aloe: A Family of Fine-tuned Open Healthcare LLMs](https://mlsanity.com/paper/2405.01886/) (2024)
- [Automated Progressive Red Teaming](https://mlsanity.com/paper/2407.03876/) (2024)
- [A Multi-Faceted Evaluation Framework for Assessing Synthetic Data Generated by Large Language Models](https://mlsanity.com/paper/2404.14445/) (2024)
- [Examining the Expanding Role of Synthetic Data Throughout the AI Development Pipeline](https://mlsanity.com/paper/2501.18493/) (2025)
- [Towards Understanding Bias in Synthetic Data for Evaluation](https://mlsanity.com/paper/2506.10301/) (2025)
- [SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domains](https://mlsanity.com/paper/2507.07229/) (2025)
- [Evaluating Alignment of Behavioral Dispositions in LLMs](https://mlsanity.com/paper/2602.11328/) (2026)
- [Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment](https://mlsanity.com/paper/2412.17417/) (2024)
- [Reasoning-Driven Synthetic Data Generation and Evaluation](https://mlsanity.com/paper/2603.29791/) (2026)
- [Evaluating the Use of Large Language Models as Synthetic Social Agents in Social Science Research](https://mlsanity.com/paper/2509.26080/) (2025)
- [Structural transparency of societal AI alignment through Institutional Logics](https://mlsanity.com/paper/2602.08246/) (2026)
- [A testable framework for AI alignment: Simulation Theology as an engineered worldview for silicon-based agents](https://mlsanity.com/paper/2602.16987/) (2026)

## Abstract (arXiv)

Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, these datasets often lack the sociotechnical nuance required for sensitive domains. We introduce NodeSynth, an evidence-grounded methodology that generates socially relevant synthetic queries by leveraging a fine-tuned taxonomy generator (TaG) anchored in real-world evidence. Evaluated against four mainstream LLMs (e.g., Claude 4.5 Haiku), NodeSynth elicited failure rates up to five times higher than human-authored benchmarks. Ablation studies confirm that our granular taxonomic expansion significantly drives these failure rates, while independent validation reveals critical deficiencies in prominent guard models (e.g., Llama-Guard-3). We open-source our end-to-end research prototype and datasets to enable scalable, high-stakes model evaluation and targeted safety interventions (https://github.com/google-research/nodesynth).