Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization

Offline reinforcement learning (RL) methods harness previous experiences to derive an optimal policy, forming the foundation for pretrained large-scale models (PLMs). When adapting to novel tasks, PLMs leverage expert trajectories as prompts to accelerate adaptation. While various prompt-tuning techniques aim to improve prompt quality, their effectiveness is often limited by initialization constraints, restricting exploration and potentially leading to suboptimal solutions. To eliminate dependence on the initial prompt, we reframe prompt-tuning as conditional generative modeling, where prompts are generated from random noise. Our proposed Prompt Diffuser employs a conditional diffusion model to generate high-quality prompts. Central to our framework is trajectory reconstruction and the seamless integration of downstream task guidance during training. Experimental results validate Prompt Diffuser's effectiveness, demonstrating strong performance in meta-RL tasks.