ASM-PPO: Asynchronous and Scalable Multi-Agent PPO for Cooperative Charging

Wireless Rechargeable Sensor Networks (WRSNs) are especially promising in large-area monitoring tasks that are previously impossible to complete by traditional Wireless Sensor Networks (WSNs). Mobile Chargers (MCs) in WRSNs are to cooperatively charge battery drained sensor nodes high efficiently and with a guarantee of sensors survival. Considering the unpredictability and high dynamics of WRSNs during the charging process, Multi-Agent Reinforcement Learning (MARL) is an attractive alternative to schedule the cooperation among MCs. However, most existing MARL methods are based on Decentralized Partially Observable Markov Decision Processes (Dec-POMDP), a general framework to describe decentralized agents making decisions at the same time step. Nevertheless, MCs in WRSNs perform charging asynchronously since the charging time of each sensor node varies. To address the problem of asynchronous behavior, we first formulate an Asynchronous Dec-POMDP (AD-POMDP). We then propose an algorithm called Asynchronous and Scalable Multi-agent Proximal Policy Optimization (ASM-PPO) that allows asynchronous learning and decision-making in AD-POMDP based on two popular multi-agent reinforcement learning methods in Dec-POMDP. Furthermore, ASM-PPO takes advantage of the translation invariance in WRSNs to avoid the huge input space dimensions caused by centralized training. The evaluation results not only indicate that our method achieves much charging efficiency and the longer lifetime of sensor nodes, but also demonstrate that ASM-PPO has advantages in terms of stability and scalability over existing methods.