Deep Q-learning for the Control of PLC-based Automated Production Systems

This paper evaluates the use of Deep Reinforcement Learning to control Programmable Logic Controller-based automated Production Systems, which are characterized by multiple end-effectors that are actuated in only one or two axes. Due to the large number of actuators of which only a few affect the processing of a workpiece at a given time, these systems are challenging to learn. In this paper, Deep Q-learning is applied to a small use case focusing on sorting workpieces by color in a simulation of such a production system. The basic algorithm is hereby compared to four commonly used extensions: Double Q-learning, Dueling Networks, Prioritized Experience Replay, and Hindsight Experience Replay. For the scope of this paper, simplifications are applied to the state and action space. While the baseline implementation of Deep Q-learning is able to correctly sort 30 previously seen workpiece combinations, it does not reliably generalize to unseen ones within 45,000 training episodes. In contrast, the algorithm using all four considered extensions is able to reliably generalize to all 81 possible workpiece combinations.