# FedLLM-Align: Feature Extraction From Heterogeneous Clients

arXiv: https://arxiv.org/abs/2510.00065 · HTML: https://mlsanity.com/paper/2510.00065/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraints](https://mlsanity.com/paper/1910.01991/) (2019)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning](https://mlsanity.com/paper/2404.09491/) (2024)
- [Large Scale Transfer Learning for Tabular Data via Language Modeling](https://mlsanity.com/paper/2406.12031/) (2024)
- [Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Features](https://mlsanity.com/paper/2204.13399/) (2022)
- [What Do We Mean by Generalization in Federated Learning?](https://mlsanity.com/paper/2110.14216/) (2021)
- [Fed2: Feature-Aligned Federated Learning](https://mlsanity.com/paper/2111.14248/) (2021)
- [PTab: Using the Pre-trained Language Model for Modeling Tabular Data](https://mlsanity.com/paper/2209.08060/) (2022)
- [PGFed: Personalize Each Client's Global Objective for Federated Learning](https://mlsanity.com/paper/2212.01448/) (2022)
- [FedXGBoost: Privacy-Preserving XGBoost for Federated Learning](https://mlsanity.com/paper/2106.10662/) (2021)
- [Classifier Clustering and Feature Alignment for Federated Learning under Distributed Concept Drift](https://mlsanity.com/paper/2410.18478/) (2024)
- [Latte: Transfering LLMs` Latent-level Knowledge for Few-shot Tabular Learning](https://mlsanity.com/paper/2505.05237/) (2025)

## Abstract (arXiv)

Federated learning (FL) enables collaborative model training without sharing raw data, making it attractive for privacy-sensitive domains, e.g., healthcare, finance, and IoT. A major obstacle, however, is the potential heterogeneity of tabular data across clients, in practical settings, where schema mismatches and incompatible feature spaces prevent straightforward aggregation. To address this challenge, this paper proposes FedLLM-Align, a federated learning framework that leverages pretrained transformer based language models for feature extraction. Towards this objective, FedLLM-Align serializes tabular records into text and derives semantically aligned embeddings from a pretrained LLM encoder, e.g, DistilBERT, facilitating lightweight local classifier heads that can be trained in a federated manner using standard aggregation schemes, e.g., FedAvg, while keeping all raw data records local. To quantify the merits and trade-offs of FedLLM-Align, we evaluate the proposed framework on binary classification tasks from two different domains: i) Coronary heart disease prediction on partitioned Framingham Heart Study data, and ii) Customer churn prediction on a financial dataset. FedLLM-Align outperforms state-of-the-art baselines by up to 25% in terms of the F1 score, under simulated schema heterogeneity, and achieves a 65% reduction in the communication overhead. These results establish FedLLM-Align as a privacy-preserving and communication-efficient approach for federated training based on clients with heterogeneous tabular datasets, commonly encountered in practice.