Execution Skill Estimation

In domains with continuous action spaces, one characteristic of an agent is their precision in executing intended actions. An agent's execution skill significantly impacts their success as it determines how much executed actions deviate from intended actions. We introduce the problem of estimating an agent's execution skill level given only observations of their executed actions. The main difficulty is that while executed actions are observed, the intended actions are not, thus the amount of action deviation due to imperfect execution skill is not obvious. We introduce a simple experimental domain in which this problem can be studied and present a method that focuses on observed rewards to estimate execution skill. This method is experimentally evaluated and shown be able to estimate an agent's execution skill under certain conditions.