ANOTO: Improving Automated Negotiation via Offline-to-Online Reinforcement Learning

Siqi Chen (School of Information Science and Engineering, Chongqing Jiaotong University), Jianing Zhao (College of Intelligence and Computing, Tianjin University), Kai Zhao (College of Intelligence and Computing, Tianjin University), Gerhard Weiss (Department of Advanced Computing Sciences, Maastricht University), Fengyun Zhang (College of Intelligence and Computing, Tianjin University), Ran Su (College of Intelligence and Computing, Tianjin University), Yang Dong (School of Information Science and Engineering, Chongqing Jiaotong University), Daqian Li (School of Information Science and Engineering, Chongqing Jiaotong University), Kaiyou Lei (College of Computer and Information Science, Southwest University)

Abstract

Automated negotiation is a crucial component for establishing cooperation and collaboration within multi-agent systems. While reinforcement learning (RL)-based negotiating agents have achieved remarkable success in various scenarios, they still face limitations due to certain assumptions on which they are based. In this work, we proposes a novel approach called ANOTO to improve the negotiating agents' ability via offline-to-online RL. ANOTO enables a negotiating agent (1) to communicate with opponents using an end-to-end strategy that covers all negotiation actions, (2) to learn negotiation strategies from historical offline data without requiring active interactions, and (3) to enhance the optimization process during the online phase, facilitating rapid and stable performance improvements for the learned offline strategies. Experimental results, based on a number of negotiation scenarios and recent winning agents from the Automated Negotiating Agents Competitions (ANAC), are provided.