# Concordia: Self-Improving Synthetic Tables for Federated LLMs

arXiv: https://arxiv.org/abs/2605.09855 · HTML: https://mlsanity.com/paper/2605.09855/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Privacy-preserving tabular data synthesis](https://mlsanity.com/synth/privacy-preserving-tabular-data-synthesis/)

## Builds on (most cited first, 24 of 24)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [The Future of Digital Health with Federated Learning](https://mlsanity.com/paper/2003.08119/) (2020)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [Towards Personalized Federated Learning](https://mlsanity.com/paper/2103.00710/) (2021)
- [TabDDPM: Modelling Tabular Data with Diffusion Models](https://mlsanity.com/paper/2209.15421/) (2022)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Classification Accuracy Score for Conditional Generative Models](https://mlsanity.com/paper/1905.10887/) (2019)
- [REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformers](https://mlsanity.com/paper/2302.02041/) (2023)
- [SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models](https://mlsanity.com/paper/2308.06522/) (2023)
- [FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models](https://mlsanity.com/paper/2310.10049/) (2023)
- [Federated Learning via Synthetic Data](https://mlsanity.com/paper/2008.04489/) (2020)
- [Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning](https://mlsanity.com/paper/2406.08527/) (2024)
- [FedSynth: Gradient Compression via Synthetic Data in Federated Learning](https://mlsanity.com/paper/2204.01273/) (2022)
- [FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Models](https://mlsanity.com/paper/2406.04845/) (2024)
- [POPri: Private Federated Learning using Preference-Optimized Synthetic Data](https://mlsanity.com/paper/2504.16438/) (2025)
- [Federated Knowledge Recycling: Privacy-Preserving Synthetic Data Sharing](https://mlsanity.com/paper/2407.20830/) (2024)
- [Is API Access to LLMs Useful for Generating Private Synthetic Tabular Data?](https://mlsanity.com/paper/2502.06555/) (2025)
- [Query, Don't Train: Privacy-Preserving Tabular Prediction from EHR Data via SQL Queries](https://mlsanity.com/paper/2505.21801/) (2025)
- [Synthetic Data Aided Federated Learning Using Foundation Models](https://mlsanity.com/paper/2407.05174/) (2024)
- [RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis](https://mlsanity.com/paper/2502.18517/) (2025)
- [Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Models](https://mlsanity.com/paper/2409.20135/) (2024)
- [LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples](https://mlsanity.com/paper/2410.19114/) (2024)
- [Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluation](https://mlsanity.com/paper/2509.20680/) (2025)

## Abstract (arXiv)

Federated learning (FL) enables training large language models (LLMs) without sharing raw data, but adapting LLMs under strict data isolation and non-IID client distributions remains challenging in practice. Synthetic data offers a natural privacy-preserving surrogate for local training, yet existing federated pipelines typically treat synthetic generation as static or loosely coupled with downstream optimization, leading to rapidly diminishing utility under heterogeneous clients. We study federated adaptation of LLMs on tabular tasks where raw records and validation data cannot be shared, and local training must rely entirely on synthetic tables. We propose Concordia, a tri-level optimization framework that aligns synthetic data generation with federated validation utility despite these constraints. At the client level, models are adapted via parameter-efficient LoRA training on synthetic tables. Clients additionally learn lightweight utility scorers from private validation feedback to reweight synthetic samples during local training. At the outer level, each client refines its own synthetic table generator using group-relative policy optimization (GRPO), guided by an ensemble of heterogeneous scorers shared across clients, without aggregating generator parameters or exposing validation data. Experiments on privacy-sensitive tabular benchmarks from finance and healthcare demonstrate that Concordia consistently improves federated performance, cross-client stability, and robustness to distribution shift compared to static and decoupled synthetic-data baselines.