Provable distributed adaptive temporal-difference learning over time-varying networks

Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as MS-ADTD) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of MS-ADTD, the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments.