Enhancing Offline Safe Reinforcement Learning with Trajectory-Constrained Diffusion Planning

Hengrui Zhang (Beijing Jiaotong University, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence), Youfang Lin (Beijing Jiaotong University, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence), Shuo Shen (Interactive Entertainment Group, Tencent), Hanfeng Lin (Beijing Jiaotong University, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence), Peng Cheng (Beijing Jiaotong University, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence), Sheng han (Beijing Jiaotong University , Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence), Kai Lv (Beijing Jiaotong University, Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence)

Abstract

Recent approaches have utilized the RL via Supervised Learning (RvS) framework to model offline safe RL. However, these methods overlook the fundamental differences between reward maximization and constraint satisfaction, treating them identically with guidance sampling, and requiring different hyperparameters for different constraint conditions. To address these limitations, we propose a novel framework, the Trajectory-Constrained Diffusion Planner (TCDP), which reframes offline safe RL as a product of trajectory conditional probabilities and energy functions. Additionally, we introduce Cost-returns-To-Go relabeling with Data Augmentation (CTGDA) and the Quantile Normalization (QN) technique, enabling the adaptation to various constraints without retraining or extensive hyperparameter adjustments.