mlsanityData to 7 Oct 2026

Concordia: Self-Improving Synthetic Tables for Federated LLMs

arXiv 2605.09855 ↗First posted 11 May 2026cs.LGcited by 0

On the map: Synthetic data and self-improvement for LLMs → Privacy-preserving tabular data synthesis

Abstract (arXiv)

Federated learning (FL) enables training large language models (LLMs) without sharing raw data, but adapting LLMs under strict data isolation and non-IID client distributions remains challenging in practice. Synthetic data offers a natural privacy-preserving surrogate for local training, yet existing federated pipelines typically treat synthetic generation as static or loosely coupled with downstream optimization, leading to rapidly diminishing utility under heterogeneous clients. We study federated adaptation of LLMs on tabular tasks where raw records and validation data cannot be shared, and local training must rely entirely on synthetic tables. We propose Concordia, a tri-level optimization framework that aligns synthetic data generation with federated validation utility despite these constraints. At the client level, models are adapted via parameter-efficient LoRA training on synthetic tables. Clients additionally learn lightweight utility scorers from private validation feedback to reweight synthetic samples during local training. At the outer level, each client refines its own synthetic table generator using group-relative policy optimization (GRPO), guided by an ensemble of heterogeneous scorers shared across clients, without aggregating generator parameters or exposing validation data. Experiments on privacy-sensitive tabular benchmarks from finance and healthcare demonstrate that Concordia consistently improves federated performance, cross-client stability, and robustness to distribution shift compared to static and decoupled synthetic-data baselines.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Privacy-preserving tabular data synthesis, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 24 of its 46 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

24 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025POPri: Private Federated Learning using Preference-Optimized Synthetic Data23 Apr
  2. 2025RewardDS: Privacy-Preserving Fine-Tuning for Large Language Models via Reward Driven Data Synthesis23 Feb

Other references · 22

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2018Learning to Reweight Examples for Robust Deep Learningcited by 670
  3. 2020The Future of Digital Health with Federated Learningcited by 625
  4. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  5. 2021Towards Personalized Federated Learningcited by 462
  6. 2022TabDDPM: Modelling Tabular Data with Diffusion Modelscited by 373
  7. 2022Language Models are Realistic Tabular Data Generatorscited by 290
  8. 2019Classification Accuracy Score for Conditional Generative Modelscited by 134
  9. 2023REaLTabFormer: Generating Realistic Relational and Tabular Data using Transformerscited by 93
  10. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  11. 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
  12. 2020Federated Learning via Synthetic Datacited by 53
10 more
  1. 2024Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoningcited by 48
  2. 2022FedSynth: Gradient Compression via Synthetic Data in Federated Learningcited by 30
  3. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 26
  4. 2024Federated Knowledge Recycling: Privacy-Preserving Synthetic Data Sharingcited by 7
  5. 2025Is API Access to LLMs Useful for Generating Private Synthetic Tabular Data?cited by 5
  6. 2025Query, Don't Train: Privacy-Preserving Tabular Prediction from EHR Data via SQL Queriescited by 5
  7. 2024Synthetic Data Aided Federated Learning Using Foundation Modelscited by 3
  8. 2024Optimizing Cross-Client Domain Coverage for Federated Instruction Tuning of Large Language Modelscited by 2
  9. 2024LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samplescited by 1
  10. 2025Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025PPC-GPT: Federated Task-Specific Compression of Large Language Models via Pruning and Chain-of-Thought Distillation
  2. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRA
  3. 2024Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data
  4. 2024FedCoT: Federated Chain-of-Thought Distillation for Large Language Models
  5. 2025FedRand: Enhancing Privacy in Federated Learning with Randomized LoRA Subparameter Updates
  6. 2024Federated In-Context LLM Agent Learning
  7. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data
  8. 2025FedSEA-LLaMA: A Secure, Efficient and Adaptive Federated Splitting Framework for Large Language Models