FedHQL: Federated Heterogeneous Q-Learning

Flint Xiaofeng Fan (National University of Singapore), Yining Ma (National University of Singapore), Zhongxiang Dai (National University of Singapore), Cheston Tan (I2R, A*STAR), Bryan Kian Hsiang Low (National University of Singapore)

Abstract

This study introduces the problem setting of Federated Reinforcement Learning with Heterogeneous And bLack-box agEnts (FedRL-HALE), in which multiple RL agents with varying policy parameterizations, training configurations, and exploration strategies work together to optimize their policies through the proposed Federated Heterogeneous Q-Learning (FedHQL) algorithm. Empirical results demonstrate the effectiveness of FedHQL in improving system performance and increasing the sample efficiency of individual agents with high confidence.