Dynamic Option Creation in Option-Critic Reinforcement Learning

The options framework introduces the concept of temporal abstraction in MDPs by combining high level courses of action with primitive, single-step actions which can greatly improve planning and learning speeds. Throughout the past two decades, there has been active interest in autonomous option discovery, as well as determining what characterizes a good option. One example of such interest and advance is the Option-Critic Architecture. However, given that the ideal number of options for learning an optimal policy is not evident for most problems, Option-Critic's reliance on a fixed set of options proves as a limitation. In the present work, we propose an algorithm for creating options dynamically in training time, using the Fast-Planning Option-Critic implementation as a base. The Dynamic Option Creation algorithm (DOC) analyzes the variance in episodic returns when selecting each option to determine whether the learning process would benefit from a new option. Our method achieves similar per-episode returns as FPOC in the four-rooms environment, with the added benefit of discovering the ideal number of options automatically.