CADP: Towards Better Centralized Learning for Decentralized Execution in MARL

Yihe Zhou (Zhejiang University), Shunyu Liu (Nanyang Technological University), Yunpeng Qing (Zhejiang University), Tongya Zheng (Big Graph Center, Hangzhou City University), Kaixuan Chen (State Key Laboratory of BC&DS, Zhejiang University), Jie Song (State Key Laboratory of BC&DS, Zhejiang University), Mingli Song (State Key Laboratory of BC&DS, Zhejiang University)

Abstract

Centralized Training with Decentralized Execution (CTDE) has recently emerged as a popular framework for cooperative Multi-Agent Reinforcement Learning (MARL), where agents can use additional global state information to guide training in a centralized way and make their own decisions only based on decentralized local policies. Despite the encouraging results achieved, CTDE makes an independence assumption on agent policies, which limits agents from adopting global cooperative information from each other during CT. Therefore, we argue that the existing CTDE framework cannot fully utilize global information for training, leading to an inefficient joint exploration and perception, which can degrade the final performance. In this paper, we introduce a novel Centralized Advising and Decentralized Pruning (CADP) framework for MARL, that not only enables an efficacious message exchange among agents during training but also guarantees DE.