# Can Textual Gradient Work in Federated Learning?

arXiv: https://arxiv.org/abs/2502.19980 · HTML: https://mlsanity.com/paper/2502.19980/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 56 references that are in the map.
- First posted on arXiv 27 Feb 2025.

## Builds on (most cited first, 30 of 50)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://mlsanity.com/paper/2306.02561/) (2023)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Improving LoRA in Privacy-preserving Federated Learning](https://mlsanity.com/paper/2403.12313/) (2024)
- [FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning](https://mlsanity.com/paper/2309.00363/) (2023)
- [OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning](https://mlsanity.com/paper/2402.06954/) (2024)
- [Large Language Models as Evolutionary Optimizers](https://mlsanity.com/paper/2310.19046/) (2023)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)

## Built on by (most cited first, 15 of 15)

- [Federated In-Context Learning: Iterative Refinement for Improved Answer Quality](https://mlsanity.com/paper/2506.07440/) (2025)
- [DoctorRAG: Medical RAG Fusing Knowledge with Patient Analogy through Textual Gradients](https://mlsanity.com/paper/2505.19538/) (2025)
- [Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions](https://mlsanity.com/paper/2506.08234/) (2025)
- [ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"](https://mlsanity.com/paper/2508.04086/) (2025)
- [Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generation](https://mlsanity.com/paper/2504.13054/) (2025)
- [TextResNet: Decoupling and Routing Optimization Signals in Compound AI Systems via Deep Residual Tuning](https://mlsanity.com/paper/2602.08306/) (2026)
- [Typed Federated Artifacts for the Agentic Web:Sharing Tool-Routing Knowledge Across Frozen,Heterogeneous LLM Agents](https://mlsanity.com/paper/2609.06815/) (2026)
- [Textual Equilibrium Propagation for Deep Compound AI Systems](https://mlsanity.com/paper/2601.21064/) (2026)
- [FERA: Uncertainty-Aware Federated Reasoning for Large Language Models](https://mlsanity.com/paper/2605.10082/) (2026)
- [FedRecon: Missing Modality Reconstruction in Heterogeneous Distributed Environments](https://mlsanity.com/paper/2504.09941/) (2025)
- [Federated Foundation Language Model Post-Training Should Focus on Open-Source Models](https://mlsanity.com/paper/2505.23593/) (2025)
- [Synapse: Federated Tool Routing via Typed Compendium Artifacts](https://mlsanity.com/paper/2602.00911/) (2026)
- [CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization](https://mlsanity.com/paper/2607.18622/) (2026)
- [Federated Agent Optimization](https://mlsanity.com/paper/2610.01195/) (2026)
- [PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?](https://mlsanity.com/paper/2610.19458/) (2026)

## Abstract (arXiv)

Recent studies highlight the promise of LLM-based prompt optimization, especially with TextGrad, which automates differentiation'' via texts and backpropagates textual feedback. This approach facilitates training in various real-world applications that do not support numerical gradient propagation or loss calculation. In this paper, we systematically explore the potential and challenges of incorporating textual gradient into Federated Learning (FL). Our contributions are fourfold. Firstly, we introduce a novel FL paradigm, Federated Textual Gradient (FedTextGrad), that allows clients to upload locally optimized prompts derived from textual gradients, while the server aggregates the received prompts. Unlike traditional FL frameworks, which are designed for numerical aggregation, FedTextGrad is specifically tailored for handling textual data, expanding the applicability of FL to a broader range of problems that lack well-defined numerical loss functions. Secondly, building on this design, we conduct extensive experiments to explore the feasibility of FedTextGrad. Our findings highlight the importance of properly tuning key factors (e.g., local steps) in FL training. Thirdly, we highlight a major challenge in FedTextGrad aggregation: retaining essential information from distributed prompt updates. Last but not least, in response to this issue, we improve the vanilla variant of FedTextGrad by providing actionable guidance to the LLM when summarizing client prompts by leveraging the Uniform Information Density principle. Through this principled study, we enable the adoption of textual gradients in FL for optimizing LLMs, identify important issues, and pinpoint future directions, thereby opening up a new research area that warrants further investigation.