Knowledge Revision for Reinforcement Learning with Abstract MDPs

Reward shaping is a method often used in RL so as to provide domain knowledge to agents and thus improve learning. An unrealistic assumption however is that the provided knowledge is always correct. This assumption can lead to poor performance in terms of total reward and convergence speed in case it is not met. Previous research demonstrated the use of plan-based reward shaping with knowledge revision in a single agent scenario where agents showed that they can quickly identify and revise erroneous knowledge and thus benefit from more accurate plans. This method however has no mechanism to deal with non-deterministic scenarios and is thus limited to deterministic domains. In this paper we present a method to provide heuristic knowledge via abstract MDPs, coupled with a revision algorithm to manage the cases where the provided domain knowledge is wrong. We show empirically that our method can efficiently revise erroneous knowledge even in the cases where the environment is non-deterministic and also removes the need for some of the assumptions present in plan-based reward shaping with knowledge revision.