# FedMatch: Federated Learning Over Heterogeneous Question Answering Data

arXiv: https://arxiv.org/abs/2108.05069 · HTML: https://mlsanity.com/paper/2108.05069/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2021.

## Built on by (most cited first, 23 of 23)

- [FedGH: Heterogeneous Federated Learning with Generalized Global Header](https://mlsanity.com/paper/2303.13137/) (2023)
- [pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning](https://mlsanity.com/paper/2310.13283/) (2023)
- [pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2402.01350/) (2024)
- [Federated Model Heterogeneous Matryoshka Representation Learning](https://mlsanity.com/paper/2406.00488/) (2024)
- [Tunable Soft Prompts are Messengers in Federated Learning](https://mlsanity.com/paper/2311.06805/) (2023)
- [FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learning](https://mlsanity.com/paper/2312.09006/) (2023)
- [CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks](https://mlsanity.com/paper/2402.16767/) (2024)
- [Can Foundation Models Help Us Achieve Perfect Secrecy?](https://mlsanity.com/paper/2205.13722/) (2022)
- [pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing](https://mlsanity.com/paper/2311.06879/) (2023)
- [C2A: Client-Customized Adaptation for Parameter-Efficient Federated Learning](https://mlsanity.com/paper/2411.00311/) (2024)
- [Advancements in Federated Learning: Models, Methods, and Privacy](https://mlsanity.com/paper/2302.11466/) (2023)
- [Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning](https://mlsanity.com/paper/2212.05789/) (2022)
- [pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learning](https://mlsanity.com/paper/2404.17847/) (2024)
- [FedRN: Exploiting k-Reliable Neighbors Towards Robust Federated Learning](https://mlsanity.com/paper/2205.01310/) (2022)
- [Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence](https://mlsanity.com/paper/2311.12358/) (2023)
- [Elastically-Constrained Meta-Learner for Federated Learning](https://mlsanity.com/paper/2306.16703/) (2023)
- [FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data](https://mlsanity.com/paper/2506.20245/) (2025)
- [Revisiting Communication-Efficient Federated Learning with Balanced Global and Local Updates](https://mlsanity.com/paper/2205.01470/) (2022)
- [PEFT-MedAware: Large Language Model for Medical Awareness](https://mlsanity.com/paper/2311.10697/) (2023)
- [FedMCP: Parameter-Efficient Federated Learning with Model-Contrastive Personalization](https://mlsanity.com/paper/2409.00116/) (2024)
- [Whom to Trust? Adaptive Collaboration in Personalized Federated Learning](https://mlsanity.com/paper/2507.00259/) (2025)
- [Social Hippocampus Memory Learning](https://mlsanity.com/paper/2603.25614/) (2026)
- [Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems](https://mlsanity.com/paper/2606.25533/) (2026)

## Abstract (arXiv)

Question Answering (QA), a popular and promising technique for intelligent information access, faces a dilemma about data as most other AI techniques. On one hand, modern QA methods rely on deep learning models which are typically data-hungry. Therefore, it is expected to collect and fuse all the available QA datasets together in a common site for developing a powerful QA model. On the other hand, real-world QA datasets are typically distributed in the form of isolated islands belonging to different parties. Due to the increasing awareness of privacy security, it is almost impossible to integrate the data scattered around, or the cost is prohibited. A possible solution to this dilemma is a new approach known as federated learning, which is a privacy-preserving machine learning technique over distributed datasets. In this work, we propose to adopt federated learning for QA with the special concern on the statistical heterogeneity of the QA data. Here the heterogeneity refers to the fact that annotated QA data are typically with non-identical and independent distribution (non-IID) and unbalanced sizes in practice. Traditional federated learning methods may sacrifice the accuracy of individual models under the heterogeneous situation. To tackle this problem, we propose a novel Federated Matching framework for QA, named FedMatch, with a backbone-patch architecture. The shared backbone is to distill the common knowledge of all the participants while the private patch is a compact and efficient module to retain the domain information for each participant. To facilitate the evaluation, we build a benchmark collection based on several QA datasets from different domains to simulate the heterogeneous situation in practice. Empirical studies demonstrate that our model can achieve significant improvements against the baselines over all the datasets.