mlsanityData to 14 Oct 2026

Can Textual Gradient Work in Federated Learning?

arXiv 2502.19980 ↗First posted 27 Feb 2025cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Recent studies highlight the promise of LLM-based prompt optimization, especially with TextGrad, which automates differentiation'' via texts and backpropagates textual feedback. This approach facilitates training in various real-world applications that do not support numerical gradient propagation or loss calculation. In this paper, we systematically explore the potential and challenges of incorporating textual gradient into Federated Learning (FL). Our contributions are fourfold. Firstly, we introduce a novel FL paradigm, Federated Textual Gradient (FedTextGrad), that allows clients to upload locally optimized prompts derived from textual gradients, while the server aggregates the received prompts. Unlike traditional FL frameworks, which are designed for numerical aggregation, FedTextGrad is specifically tailored for handling textual data, expanding the applicability of FL to a broader range of problems that lack well-defined numerical loss functions. Secondly, building on this design, we conduct extensive experiments to explore the feasibility of FedTextGrad. Our findings highlight the importance of properly tuning key factors (e.g., local steps) in FL training. Thirdly, we highlight a major challenge in FedTextGrad aggregation: retaining essential information from distributed prompt updates. Last but not least, in response to this issue, we improve the vanilla variant of FedTextGrad by providing actionable guidance to the LLM when summarizing client prompts by leveraging the Uniform Information Density principle. Through this principled study, we enable the adoption of textual gradients in FL for optimizing LLMs, identify important issues, and pinpoint future directions, thereby opening up a new research area that warrants further investigation.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 50 of its 56 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

50 of 56 references.

References · 50

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
38 more
  1. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2023Large Language Models as Optimizerscited by 864
  7. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  8. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  9. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  10. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  11. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  12. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  13. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  14. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 161
  15. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 149
  16. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  17. 2023Large Language Models as Evolutionary Optimizerscited by 124
  18. 2020If beam search is the answer, what was the question?cited by 116
  19. 2020Byzantine-Resilient Secure Federated Learningcited by 93
  20. 2019Accelerating Federated Learning via Momentum Gradient Descentcited by 76
  21. 2024FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Modelcited by 74
  22. 2020Fast-Convergent Federated Learning with Adaptive Weightingcited by 67
  23. 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
  24. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  25. 2021Fed2: Feature-Aligned Federated Learningcited by 42
  26. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  27. 2022Privately Fine-Tuning Large Language Models with Differential Privacycited by 39
  28. 2024Are Large Language Models Good Prompt Optimizers?cited by 37
  29. 2024Advances and Open Challenges in Federated Foundation Modelscited by 36
  30. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  31. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  32. 2021Federated Learning for Face Recognition with Gradient Correctioncited by 24
  33. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  34. 2023FedSoup: Improving Generalization and Personalization in Federated Learning via Selective Model Interpolationcited by 14
  35. 2022Backdoor Attack and Defense in Federated Generative Adversarial Network-based Medical Image Synthesiscited by 12
  36. 2024Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Modelscited by 8
  37. 2023Federated Learning on Virtual Heterogeneous Data with Local-global Distillationcited by 6
  38. 2024Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 8
  2. 2025DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradientscited by 5
  3. 2025Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directionscited by 4
  4. 2025ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"cited by 4
  5. 2025Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generationcited by 3
  6. 2026TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuningcited by 3
  7. 2026Typed Federated Artifacts for the Agentic Web:Sharing Tool-Routing Knowledge Across Frozen,Heterogeneous LLM Agentscited by 2
  8. 2026Textual Equilibrium Propagation for Deep Compound AI Systemscited by 1
  9. 2026FERA: Uncertainty-Aware Federated Reasoning for Large Language Modelscited by 1
  10. 2025FedRecon: Missing Modality Reconstruction in Heterogeneous Distributed Environmentscited by 0
  11. 2025Federated Foundation Language Model Post-Training Should Focus on Open-Source Modelscited by 0
  12. 2026Synapse: Federated Tool Routing via Typed Compendium Artifactscited by 0
3 more
  1. 2026CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimizationcited by 0
  2. 2026Federated Agent Optimizationcited by 0
  3. 2026PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?cited by 0