Provably Efficient Convergence of Primal-Dual Actor-Critic with Nonlinear Function Approximation

We study the convergence of the actor-critic algorithm with nonlinear function approximation under a nonconvex-nonconcave primaldual formulation. Stochastic gradient descent ascent is applied with an adaptive proximal term for robust learning rates. We show the first efficient convergence result with primal-dual actor-critic with a convergence rate of O √︃ ln(𝑁 𝑑𝐺 2) 𝑁 under Markovian sampling, where 𝐺 is the element-wise maximum of the gradient, 𝑁 is the number of iterations, and 𝑑 is the dimension of the gradient. Our result is presented with only the Polyak-Łojasiewicz (PL) condition for the dual variable, which is easy to verify and applicable to a wide range of RL scenarios.