Distributional Knowledge Transfer for Heterogeneous Federated Learning
Publication Name
Proceedings - 20th IEEE International Symposium on Parallel and Distributed Processing with Applications, 12th IEEE International Conference on Big Data and Cloud Computing, 12th IEEE International Conference on Sustainable Computing and Communications and 15th IEEE International Conference on Social Computing and Networking, ISPA/BDCloud/SocialCom/SustainCom 2022
Abstract
Federated learning (FL) produces an effective global model by aggregating multiple client weights trained on their private data. However, it is common that the data are not independently and identically distributed (non-IID) across different clients, which greatly degrades the performance of the global model. We observe that existing FL approaches mostly ignore the distribution information of client-side private data. Actually, the distribution information is a kind of structured knowledge about the data itself, and it also represents the mutual clustering relations of data examples. In this work, we propose a novel approach, namely Federated Distribution Knowledge Transfer (FedDKT), that alleviates heterogeneous FL by extracting and transferring the distribution knowledge from diverse data. Specifically, the server learns a lightweight generator to generate data and broadcasts it to the sampled clients, FedDKT decouples the feature representations of the generated data and transfers the distribution knowledge to assist model training. In other words, we exploit the similarity and shared parts of the generated data and local private data to improve the generalization ability of the FL global model and promote representation learning. Further, we also propose the similarity measure and attention measure strategies, which implement FedDKT by capturing the correlations and key dependencies among data examples, respectively. The comprehensive experiments demonstrate that FedDKT significantly improves the performance and convergence rate of the FL global model, especially when the data are extremely non-IID. In addition, FedDKT is also effective when the data are identically distributed, which fully illustrates the generalization and effectiveness of the distribution knowledge.
Open Access Status
This publication is not available as open access
First Page
747
Last Page
754
Funding Number
61672415
Funding Sponsor
National Natural Science Foundation of China
Link to publisher version (DOI)
http://dx.doi.org/10.1109/ISPA-BDCloud-SocialCom-SustainCom57177.2022.00101