关键词: 柔性作业车间动态调度, 深度强化学习, 启发式调度规则, 优先经验回放, 自适应调度

Abstract: The growing speed of economic globalization has exposed the manufacturing industry to fierce market competition and a changeable production environment. Therefore, responding effectively to the challenges posed by various dynamic events has become a key issue affecting the survival of businesses. In recent years, many enterprises have been focusing on flexible manufacturing models. In this context, the dynamic flexible job-shop scheduling problem (DFJSP) has attracted extensive attention from industry and academia.
Random events associated with workpieces are one of the reasons for driving dynamic scheduling. One of the typical problems is the dynamic flexible job-shop scheduling problem with random job arrivals (DFJSP-RJA). To address this problem, this paper develops a mixed-integer programming model in conjunction with green manufacturing propositions. The optimization objective is to simultaneously minimize the total production delay and energy consumption that includes the machine idling and processing energy consumption.
Since deep reinforcement learning (DRL) can achieve both high quality and high response in dynamic environments, and scheduling rules are widely used as an immediate response method in the pre-study of dynamic scheduling problems, this paper combines the two and proposes an algorithm, called DDQN-ST9, based on composite scheduling rules and DRL. First, based on the optimization objectives of timely completion and energy saving, six production state feature indicators with values between [0,1] are set and three scheduling rules are designed for procedure and machine scheduling respectively, which are used for the construction of feature vectors and action spaces in the later usage of the algorithm. And then the prioritized experience replay, called Sum Tree, is introduced on the basis of the DDQN algorithm to accelerate the convergence speed and improve the training efficiency. DFJSP-RJA can be regarded as a Markov decision process (MDP), in which the agent, after a perturbation occurs, selects the most suitable scheduling rule among nine composite scheduling rules by integrating the information of the current production state to complete the scheduling of the original and afterwards arrive jobs.
In order to comprehensively test the performance of the DDQN-ST9 in solving the dynamic energy-saving scheduling problem for flexible job-shop, the algorithm is simulated using several benchmark examples from the Kacem and Brandimarte series. Firstly, the nine composite scheduling rules proposed in this paper are compared with the five classical scheduling rules appearing in the literature, and the Kacem and Brandimarte benchmark instances of different sizes are solved respectively, which verifies the superiority of DDQN-ST9 in the aspects of scheduling rule design, scheduling algorithm design and improving algorithm structure. Secondly, by varying both parameters of the delivery urgency factor and the exponential distribution followed by the random arrivals of the jobs, a number of Brandimarte benchmark instances with different delivery requirements and market demands are solved, verifying that the DDQN-ST9 algorithm can effectively cope with a variety of production environment configurations.
This paper focuses on combining DRL with scheduling rules to apply to the dynamic energy-saving scheduling problem in a flexible job shop, which can be extended to other shop environments in the future, and to consider the impact of other dynamic events on production scheduling, such as machining time changes, machine failures, and so on. In addition, it can be investigated how to better optimize the solution process of complex dynamic scheduling problems based on deep reinforcement learning.

Key words: dynamic flexible job shop scheduling problem, deep reinforcement learning, heuristic scheduling rules, prioritized experience replay, adaptive scheduling