关键词: 类风湿关节炎, 马尔可夫决策过程, 实验室指标, 中药

Abstract: Rheumatoid Arthritis (RA), as a highly disabling disease requiring lifelong treatment, brings people not only great pain physically and mentally, but also poses a serious economic burden to families and society. Currently, there are about 5 million RA patients in China, but the number of doctors in the department of rheumatism is seriously insufficient. Therefore, it is of important practical significance to find the optimal treatment plan for RA from the electronic medical records of RA patients. In addition, during the treatment process of RA, as the patient's health status is not fully exposed, doctors often make multi-stage decisions in an uncertain environment, Markov Decision Process (MDP) is very suitable for modeling in uncertain environments, and currently RA patients can only control the development of their condition through lifelong treatment. Therefore, this paper proposes to apply the MDP model to the treatment process of RA patients under the infinite horizon average criterion. The theoretical significance of this paper is to provide a theoretical method and analytical steps based on historical medical data for research on hospital treatment decision-making, and the practical significance is that the treatment policies obtained through the constructed MDP model can provide reference for treatment policies for RA patients in other hospitals.
The clinical data used in this paper are from the electronic medical records of patients in the Rheumatology Department of the First Affiliated Hospital of Anhui University of Chinese Medicine. When constructing the MDP model, this paper gives the definition of each parameter one by one and uses clinical data from the electronic medical records of RA patients for inference. Firstly, this paper takes the time point at which doctors give treatment plans as the decision-making time. Secondly, this paper uses the K-modes clustering algorithm with different numbers of clusters to cluster the laboratory indexes of patients as feature variables, and ultimately obtain hidden health states which are relatively reasonable and easy to explain. Then, as traditional Chinese medicine is the basic method of clinical treatment in traditional Chinese medicine, this paper regards traditional Chinese medicine which is used by patients between two laboratory indexes tests as the basis for the action of the MDP model. Next, for the transition probability, this paper calculates the empirical transition probability based on the ratio of the number of people who take an action in a certain state and transfer to another state to the total number of people who take that action in that state, and replaces it. Finally, this paper considers the sum of the improvement degree of patients' indexes and the length of hospital stay between two laboratory indexes tests as treatment reward and treatment cost respectively.
When solving the optimal strategy of the MDP model, this paper uses relative value iteration algorithm to solve and obtains the corresponding treatment strategy, treatment reward and treatment cost. The experimental results show that the treatment reward obtained by the MDP model constructed in this paper is higher than that of the hospital, and the treatment cost is lower than that of the hospital. Therefore, it has a certain clinical application value to apply MDP model in the treatment of RA in traditional Chinese medicine.
In future work, other treatment methods such as fumigation, massage, acupuncture and moxibustion and western medicine can be incorporated into the action of MDP model, and the treatment process can also be modeled using partially observable Markov decision processes. In addition, in the process of completing this paper, we have received much support. We would like to thank Professor XIE Jingui of Technical University of Munich in Germany for putting forward this research question, and the Rheumatology Department of the First Affiliated Hospital of Anhui University of Chinese Medicine for providing electronic medical record data of RA patients.

Key words: Rheumatoid Arthritis, Markov Decision Process, laboratory indexes, traditional Chinese medicine