Deeper Understanding of Vague Instructions through Simulated Execution

A commonsense understanding of the physical world will be crucial for the robots of the future as they strive to perform everyday activities and instructions formulated by human users in natural language. One mechanism that is believed to assist human cognition in commonsense reasoning is mental simulation, the envisioning of actions before they are performed. We therefore present a system integrating simulation of robot plans with probabilistic reasoning about natural-language instructions, to create a complete pipeline from instruction to execution to storing and analyzing results of the simulation. This integration allows the robotic system to efficiently infer knowledge about the physical world that would be tedious to specify by hand in a collection of logical statements. Our system will be available online 1 for open use by researchers.