Swarm-Assisted Distributed Gradient Methods with Compressed Communication for Large-Scale Learning Over Networks

Abstract

Large-scale learning over networks increasingly relies on distributed gradient methods executed by many simple agents connected through sparse communication topologies. In such systems, communication of high-dimensional model parameters becomes a major bottleneck, while heterogeneous data and unreliable links complicate global coordination. Swarm-inspired coordination mechanisms provide an additional interaction layer where agents adapt their search directions using locally observed performance and neighborhood behavior. At the same time, compressed communication techniques reduce message sizes by sparsifying or quantizing exchanged information, trading accuracy for controlled stochastic distortion. Combining these two elements gives rise to swarm-assisted distributed gradient procedures that remain applicable when network resources and local memory are constrained. This work develops a modeling framework for such methods, focusing on the interaction between swarm dynamics, gradient tracking, and lossy inter-agent communication. The agents are viewed as nodes of a graph that maintain local parameter vectors, exchange compressed messages with neighbors, and adjust their updates based on swarm feedback signals. The resulting algorithms exploit neighborhood structure and swarm attraction to guide the joint optimization trajectory, while compression reduces overall communication load and power consumption. Analytical developments characterize how network topology, step sizes, swarm gains, and compression accuracy interact to shape error propagation and steady-state bias. The framework accommodates strongly convex and nonconvex learning objectives, illustrates tradeoffs between compression level and convergence speed, and clarifies robustness properties under gradient noise. This analysis supports the design of scalable protocols for training high-dimensional models over large, sparsely connected swarms of learning agents with limited communication budgets.