Compressed Regression over Adaptive Networks
Not on the map. Why?
Abstract (arXiv)
In this work we derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem. Agents employ the recently proposed ACTC (adapt-compress-then-combine) diffusion strategy, where the signals exchanged locally by neighboring agents are encoded with randomized differential compression operators. We provide a detailed characterization of the mean-square estimation error, which is shown to comprise a term related to the error that agents would achieve without communication constraints, plus a term arising from compression. The analysis reveals quantitative relationships between the compression loss and fundamental attributes of the distributed regression problem, in particular, the stochastic approximation error caused by the gradient noise and the network topology (through the Perron eigenvector). We show that knowledge of such relationships is critical to allocate optimally the communication resources across the agents, taking into account their individual attributes, such as the quality of their data or their degree of centrality in the network topology. We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents. Illustrative examples show that a significant performance improvement, as compared to a blind (i.e., uniform) resource allocation, can be achieved by optimizing the allocation by means of the provided mean-square-error formulas.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 22 of its 58 references that are in the map. First posted on arXiv 7 Apr 2023.
Builds on
22 of 58 references.
References · 22
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2021Convex optimizationcited by 2,479
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2018Sparsified SGD with Memorycited by 373
- 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
- 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 199
- 2019Decentralized Deep Learning with Arbitrary Communication Compressioncited by 109
- 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
- 2020Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guaranteescited by 45
- 2015Network Lasso: Clustering and Optimization in Large Graphscited by 42
- 2020A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!cited by 41
- 2018An Exact Quantized Decentralized Gradient Descent Algorithmcited by 34
10 more
- 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
- 2020Multitask learning over graphs: An Approach for Distributed, Streaming Machine Learningcited by 26
- 2013On the Learning Behavior of Adaptive Networks - Part I: Transient Analysiscited by 20
- 2021Finite-Bit Quantization For Distributed Algorithms With Linear Convergencecited by 15
- 2019Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimizationcited by 11
- 2019Semi-supervised Learning in Network-Structured Data via Total Variation Minimizationcited by 10
- 2013On the Learning Behavior of Adaptive Networks - Part II: Performance Analysiscited by 9
- 2015On the Minimax Risk of Dictionary Learningcited by 7
- 2022Quantization for decentralized learning under subspace constraintscited by 7
- 2021Distributed Adaptive Learning Under Communication Constraintscited by 4
Built on by
Citing papers per year: 2024: 2.
2 papers.