Modeling Dynamic Environments with Scene Graph Memory

Andrey Kurenkov (Stanford University), Michael Lingelbach (Stanford University), Tanmay Agarwal (Stanford University), Chengshu Li (Stanford University), Emily Jin (Stanford University), Ruohan Zhang (Stanford University), Li Fei-Fei (Stanford University), Jiajun Wu (Stanford University), Silvio Savarese (Salesforce AI Research), Roberto Martín-Martín (University of Texas at Austin)

Abstract

Embodied AI agents operating in dynamic environments often need to predict object locations to make informed decisions. We propose a method for doing this via link prediction on partially observable dynamic graphs. We represent the agent's accumulated set of observations in a data structure called a Scene Graph Memory (SGM), combine this data structure with a neural net architecture we call Node Edge Predictor (NEP), and show that it can be trained to predict the locations of objects in a variety of environments with diverse object movement dynamics. To evaluate our method, we implement the Dynamic Household Simulator, a novel benchmark which enables sampling of diverse dynamic scene graphs that follow the semantic patterns typically seen at peoples' homes. We demonstrate that our method outperforms baselines both in terms of quickly adapting to the dynamics of a new scene and in terms of its overall accuracy.