Modelling the Dynamic Joint Policy of Teammates with Attention Multi-agent DDPG

Modelling teammates' policies in cooperative multi-agent systems has long been an interest and also a big challenge for the reinforcement learning (RL) community. The interest lies in the fact that if the agent knows the teammates' policies, it can adjust its own policy accordingly to arrive at proper cooperations; while the challenge is that the agents' policies are changing continuously because they are learning concurrently to adapt to each other. In this paper, we present ATTention Multi-Agent Deep Deterministic Policy Gradient (ATT-MADDPG) to address this challenge. ATT-MADDPG extends DDPG, a single-agent actor-critic RL method, with two special designs. First, as a necessary step to model the teammates' policies, the agent should get access to the observations and actions of teammates. ATT-MADDPG adopts a centralized critic to collect such information. Second, ATT-MADDPG further enhances the centralized critic with an attention mechanism in a principled way. This attention mechanism introduces a special structure to explicitly model the dynamic joint policy of teammates in an adaptive manner, making sure that the collected information can be processed in an effective way. As a result, all agents will cooperate with each other efficiently. We evaluate our method on both benchmark tasks and the real-world packet routing tasks. Results show that ATT-MADDPG not only outperforms the state-of-the-art RL-based and rule-based methods by a large margin, but also achieves better scalability and robustness.