Provable distributed adaptive temporal-difference learning over time-varying networks

Publication Name

Expert Systems with Applications


Multi-agent reinforcement learning (MARL) has been successfully applied in many fields. In MARL, the policy evaluation problem is one of crucial problems. In order to solve this problem, distributed Temporal-Difference (TD) learning algorithm is one of the most popular methods in a cooperative manner. Despite its empirical success, however, the theory of the adaptive variant of distributed TD learning still remain limited. To fill this gap, we propose an adaptive distributed temporal-difference algorithm (referred to as MS-ADTD) under Markovian sampling over time-varying networks. Furthermore, we rigorously analyze the convergence of MS-ADTD, the theoretical results show that the local estimation can converge linearly to the optimal neighborhood. Meanwhile, the theoretical results are verified by simulation experiments.

Open Access Status

This publication is not available as open access



Article Number


Funding Number


Funding Sponsor

National Natural Science Foundation of China



Link to publisher version (DOI)