Adaptive Offline Data Replay in Offline-to-Online Reinforcement Learning

Offline-to-online reinforcement learning combines the advantages of offline data utilization with online exploration to enhance sample efficiency and performance. A primary challenge lies in managing the distribution shift between offline and online data, which significantly impacts training effectiveness. Existing methods often employ fixed mixing ratios for data replay, but these require taskspecific tuning and may fail to generalize across different environments. To address this, we introduce a metric that evaluates policy quality relative to offline and online data, and propose a banditbased strategy to adjust the mixing ratio adaptively, optimizing policy quality during training. Experiments across diverse environments demonstrate that our approach outperforms static methods, offering robust adaptability and minimizing manual tuning.