Multi-Advisor Dynamic Decision Making

Being able to infer the ground truth from the answers of multiple imperfect advisors is a problem of crucial importance in many decision-making applications, such as lending, trading, investment, and crowd-sourcing. It is important to make multiple decisions over time in a sequential decision-making setting. Crucially, we assume no access to ground truth and no prior knowledge about the reliability of advisers. Specifically, our research considers how to (1) learn the trustworthiness of advisers dynamically without prior information by asking multiple advisers and (2) make optimal decisions without access to the ground truth and improve this over time. To address these problems, we proposed a new method, which combines the Bayesian Weighted Voting ensemble method and Subjective Logic. It can aggregate binary answers from multiple imperfect advisors for truth inference and model the trustworthiness of advisors. We address two problems based on our method. The first is a multi-trainer interactive reinforcement learning system; the second is multi-advisor dynamic binary decision-making by maximizing the utility. The experimental results show that our approach outperforms other state-of-the-art methods.