关键词: 订单接受, 动态决策, 深度强化学习, 柔性流水车间

Abstract: Cloud manufacturing is a new intelligent manufacturing model which uses network and service platform to provide all kinds of on-demand manufacturing services for customer needs. The main characteristics of cloud manufacturing can be summarized as customer-centric, service uncertainty, and service on demand. After production enterprises participate in cloud manufacturing, there are two types of orders: established existing orders and dynamic arrival cloud orders.
In the cloud manufacturing environment, the OAS problem with the flexible flow shop as the processing environment is described as follows: After the platform sends the cloud order to the enterprise with surplus capacity, the enterprise needs to choose whether to accept the order and complete the production arrangement under the premise of producing the existing order. Order arrival follows Poisson distribution, and each order includes quantity, price, delivery time, machining part number and other information. If a cloud order is dynamically distributed to the enterprise, the enterprise needs to combine the production information of the cloud order, the production situation of the workshop and the arrival of future orders, and determine the collection of accepted orders and the production and processing arrangement, so as to maximize the total profit of the enterprise.
Based on flexible flow shop, an improved DQN algorithm is proposed to solve the problem with order acceptance and scheduling in cloud platform dynamic order dispatching. The single agent aims at maximum profit, and the single agent aims at minimum delay time and minimum disturbance. Since the objective functions of the two agents are different, they are non-homogeneous agents, so each agent adopts an independent DQN algorithm, and a dynamic interaction mechanism is established between agents. After the cloud order arrives, the receiving agent chooses to accept or reject the order and transmits the accepted order information to the placing agent. After trying different scheduling rules, the scheduling agent finds the optimal scheduling strategy through the feedback obtained by the reward function. The DQN network structure is improved in the scheduling agent, which increases the number of rules for selecting work piece and machine to 50, and further designs the process candidate set and machine candidate set combining the critical path, and the algorithm improvement strategy such as the earliest start of the process.
The improved DQN algorithm is compared with the heuristic rule, Q-learning algorithm and DQN algorithm. The numerical experiments show that the improved algorithm is stable and superior to other algorithms in terms of maximum and average profit under different delay penalty factors, with higher order acceptance rate and balanced machine load. When the number of cloud orders increases, the worst solution of the improved algorithm is also better than the other algorithms. This shows the effectiveness of the improved algorithm. The scheduling strategy of the agent can optimize the scheduling of dynamically arrived cloud orders, improve the resource utilization rate of the workshop while producing existing orders normally, and improve the profit and order acceptance rate of enterprises. According to the research, most heuristic rules are short-sighted, but they have better performance when combined with DQN algorithm. Different rules are applicable to different scheduling targets and production environments. When deciding whether to accept cloud orders, DQN algorithm, after continuous learning, chooses appropriate scheduling rules and utilizes improved strategies, and can make each rule adjust order acceptance strategy and scheduling strategy in a short time, reduce workshop disturbance, and reduce the impact of delay penalty cost on profits, so as to ensure that the enterprise can obtain the maximum profit.

Key words: order acceptance, dynamic scheduling, reinforcement learning, flexible flow shop