PNNL

Abstract

With the promising results of Deep Reinforcement Learning (DRL) based control strategies in power systems, it is imperative to adopt faster and more efficient techniques for DRL policy learning. Some state-of-the-art derivative-free DRL algorithms, which show better computational efficiency and faster convergence, can even take several hours to train an efficient DRL policy for large (or bulk) power systems. It is found that the computational burden imposed by the conventional simulators in power systems creates the main bottleneck in further reduction of the training time. We replaced the conventional power system simulator with a trained Deep Neural Network (DNN)-based surrogate model and proposed a novel model-based DRL framework leveraging imitation learning-based warm-start in policy search. However, DRL policy learning with a surrogate model is not trivial and solely depends on the accuracy of the trained surrogate model in approximating power system dynamics. We mitigate these limitations by (a) utilizing multi-step prediction loss to improve the generalization capability of the surrogate model and (b) updating the surrogate model in the training phase of DRL policy. The proposed method is tested with the IEEE-300 bus system, and it achieves the desired control performance with a reduction of 87.7% of the training time compared to its model-free counterparts.