An Online Human-Agent Interaction System: A Brain-controlled Agent Playing Games in Unity

Human-agent interactions present people guide an object or agent to act as human intentions. This demonstration work develops an online human-agent interaction system, particularly targeting the brain-computer interface (BCI), which uses real-time brain cortex signals: electroencephalogram (EEG) to control the agent in Unity3D game platform. The developed system also provides the online visualisation of EEG signals, including pre-processed temporal data and power spectral in three frequency bands (theta, alpha, and beta). To build this systematic work, we firstly collect wireless EEG signals via the Bluetooth transmission from a commercially available 14-channel brainware headset (Emotiv). EEG signals are then pre-processed and fed into a trained deep learning model to predict the human intentions, which will be sent to Unity3D platform to control an agent's movements in game playing, such as a karting game scenario. The online testing results show the feasibility of our systematic work that will benefit for human-agent interaction community. The demonstration video can be viewed at the following link: https://youtu.be/9AWKHeatc6I