FedLLM-Align: Feature Extraction From Heterogeneous Clients

arXiv 2510.00065 ↗First posted 29 Sep 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) enables collaborative model training without sharing raw data, making it attractive for privacy-sensitive domains, e.g., healthcare, finance, and IoT. A major obstacle, however, is the potential heterogeneity of tabular data across clients, in practical settings, where schema mismatches and incompatible feature spaces prevent straightforward aggregation. To address this challenge, this paper proposes FedLLM-Align, a federated learning framework that leverages pretrained transformer based language models for feature extraction. Towards this objective, FedLLM-Align serializes tabular records into text and derives semantically aligned embeddings from a pretrained LLM encoder, e.g, DistilBERT, facilitating lightweight local classifier heads that can be trained in a federated manner using standard aggregation schemes, e.g., FedAvg, while keeping all raw data records local. To quantify the merits and trade-offs of FedLLM-Align, we evaluate the proposed framework on binary classification tasks from two different domains: i) Coronary heart disease prediction on partitioned Framingham Heart Study data, and ii) Customer churn prediction on a financial dataset. FedLLM-Align outperforms state-of-the-art baselines by up to 25% in terms of the F1 score, under simulated schema heterogeneity, and achieves a 65% reduction in the communication overhead. These results establish FedLLM-Align as a privacy-preserving and communication-efficient approach for federated training based on clients with heterogeneous tabular datasets, commonly encountered in practice.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 24 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

16 of 24 references.

References · 16

2018Federated Optimization in Heterogeneous Networkscited by 2,971
2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
2020Knowledge Distillation: A Surveycited by 1,578
2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
2024Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learningcited by 79
2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 73
2022Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Featurescited by 63
2021What Do We Mean by Generalization in Federated Learning?cited by 62
2021Fed2: Feature-Aligned Federated Learningcited by 42
2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30

4 more

2022PGFed: Personalize Each Client's Global Objective for Federated Learningcited by 13
2021FedXGBoost: Privacy-Preserving XGBoost for Federated Learningcited by 9
2024Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Driftcited by 8
2025Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learningcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way