Loss of Plasticity: A New Perspective on Solving Multi-Agent Exploration for Sparse Reward Tasks

Exploration remains a fundamental yet challenging problem in Multi-Agent Reinforcement Learning (MARL). In this paper, we address the issue from a novel perspective: the loss of plasticity, a phenomenon characterized by the declining adaptability of neural networks to adapt to new trajectories as training progresses. Through systematic empirical studies, we derive several key insights: (1) Plasticity loss is widespread in MARL; (2) Without timely interventions to restore plasticity, neural networks struggle to learn effective exploration strategies, even when provided with novel and informative data; (3) While restoring plasticity can enhance learning capabilities and exploration efficiency, the process is inherently unstable, with its effectiveness largely depending on which modules are restored and the timing of the intervention. Based on these findings, we propose Plasticity-Aware Multi-Agent Exploration (PAME), which introduces targeted and minimal interventions to enhance plasticity in specific modules of MARL at optimal times. Our results show that PAME consistently outperforms state-of-the-art methods in terms of exploration efficiency.