Abstract: Rheumatoid Arthritis (RA), as a highly disabling disease requiring lifelong treatment, brings people not only great pain physically and mentally, but also poses a serious economic burden to families and society. Currently, there are about 5 million RA patients in China, but the number of doctors in the department of rheumatism is seriously insufficient. Therefore, it is of important practical significance to find the optimal treatment plan for RA from the electronic medical records of RA patients. In addition, during the treatment process of RA, as the patient's health status is not fully exposed, doctors often make multi-stage decisions in an uncertain environment, Markov Decision Process (MDP) is very suitable for modeling in uncertain environments, and currently RA patients can only control the development of their condition through lifelong treatment. Therefore, this paper proposes to apply the MDP model to the treatment process of RA patients under the infinite horizon average criterion. The theoretical significance of this paper is to provide a theoretical method and analytical steps based on historical medical data for research on hospital treatment decision-making, and the practical significance is that the treatment policies obtained through the constructed MDP model can provide reference for treatment policies for RA patients in other hospitals.
The clinical data used in this paper are from the electronic medical records of patients in the Rheumatology Department of the First Affiliated Hospital of Anhui University of Chinese Medicine. When constructing the MDP model, this paper gives the definition of each parameter one by one and uses clinical data from the electronic medical records of RA patients for inference. Firstly, this paper takes the time point at which doctors give treatment plans as the decision-making time. Secondly, this paper uses the K-modes clustering algorithm with different numbers of clusters to cluster the laboratory indexes of patients as feature variables, and ultimately obtain hidden health states which are relatively reasonable and easy to explain. Then, as traditional Chinese medicine is the basic method of clinical treatment in traditional Chinese medicine, this paper regards traditional Chinese medicine which is used by patients between two laboratory indexes tests as the basis for the action of the MDP model. Next, for the transition probability, this paper calculates the empirical transition probability based on the ratio of the number of people who take an action in a certain state and transfer to another state to the total number of people who take that action in that state, and replaces it. Finally, this paper considers the sum of the improvement degree of patients' indexes and the length of hospital stay between two laboratory indexes tests as treatment reward and treatment cost respectively.
When solving the optimal strategy of the MDP model, this paper uses relative value iteration algorithm to solve and obtains the corresponding treatment strategy, treatment reward and treatment cost. The experimental results show that the treatment reward obtained by the MDP model constructed in this paper is higher than that of the hospital, and the treatment cost is lower than that of the hospital. Therefore, it has a certain clinical application value to apply MDP model in the treatment of RA in traditional Chinese medicine.
In future work, other treatment methods such as fumigation, massage, acupuncture and moxibustion and western medicine can be incorporated into the action of MDP model, and the treatment process can also be modeled using partially observable Markov decision processes. In addition, in the process of completing this paper, we have received much support. We would like to thank Professor XIE Jingui of Technical University of Munich in Germany for putting forward this research question, and the Rheumatology Department of the First Affiliated Hospital of Anhui University of Chinese Medicine for providing electronic medical record data of RA patients.

Key words: Rheumatoid Arthritis, Markov Decision Process, laboratory indexes, traditional Chinese medicine

摘要： 类风湿关节炎(RA)不仅给人们带来了巨大的身心痛苦,同时也带来了巨大的成本。针对RA的治疗过程,本文提出将马尔可夫决策过程(MDP)应用于该过程中。对于建立MDP所需的各个参数,本文给出定义方式并利用临床数据进行推断。首先本文利用患者的实验室指标来衡量健康状态,然后将患者使用的中药视为行动的基础,接着分别将患者指标的改善程度之和与患者两次实验室指标检查之间已住院的时长视为治疗报酬与治疗成本,最后利用相对值迭代算法求解并得到了相应的治疗策略以及治疗报酬与治疗成本。实验结果表明,本文所得到的治疗报酬要高于医院的报酬且治疗成本要低于医院的成本,将MDP模型用于RA的中医治疗中具有一定的临床应用价值。

关键词: 类风湿关节炎, 马尔可夫决策过程, 实验室指标, 中药