Can you see how I learn? Human Observers' Inferences about Reinforcement Learning Agents' Learning Processes

Human-in-the-loop Reinforcement Learning (RL) often suffers from suboptimal human teaching signals. Yet, how humans perceive and interpret RL agent's learning behavior is largely unknown. In a bottom-up approach with two experiments, this work provides a data-driven understanding of the factors in RL agents' behavior that influence the understanding of the agent's learning process for human observers. In two consecutive experiments with two different RL agents (a tabular and function approximation agent in a navigation and a manipulation task), human observations of agent learning behavior was assessed and systematically analyzed. Four common emerging themes were observed: Agent Goals, Knowledge, Decision Making and Learning Mechanisms, each with specific subclusters, offering insights for transparency in RL and HRI.