Towards Scalable and Robust Decision Making in Partially Observable, Multi-Agent Environments

Designing autonomous agents that can interact effectively with other agents is an important problem in multi-agent systems. For real-world applications these agents must also be able to handle partial observability and scale to complex environments. We present two efficient planning algorithms for multi-agent, partially observable environments. The first, Interactive Nested Tree Monte-Carlo Planning (I-NTMCP), is a novel extension of Monte-Carlo Tree Search (MCTS) to Interactive Partially Observable Markov Decision Processes (I-POMDPs). Compared to existing methods, I-NTMCP is able to scale to significantly larger I-POMDP problems while modelling the other agent to deeper reasoning levels. The second algorithm, Bayes-Adaptive Partially Observable Stochastic Game Monte-Carlo Planning (BA-POSGMCP), combines a novel metapolicy with MCTS for scalable type-based reasoning. Through comprehensive empirical analysis in large cooperative, competitive and mixed domains we demonstrate that BA-POSGMCP is able to more effectively adapt online to diverse sets of agents in larger problems than previous methods. To support further research we have also developed POSGGym, an open-source library of multi-agent, partially observable environments supporting both planning and learning methods, along with POSGGym-Agents, a suite of policies for these environments.