Interrater agreement and diagnostic accuracy for pattern differentiation and acupoint prescription

Abstract

Introduction: Traditional Chinese medicine (TCM) has been developed for at least 3,000 years for a geographic community, socially and culturally distant from the West. The physiology of TCM is based on the vital substances in the internal organs and channels and collateral acupoints. Clinical manifestations of an individual are obtained during their evaluation through inspection (Ip), auscultation, olfaction (AO), inquiry (Iq) and palpation (P) - collectively known as the four tests, where the signs and symptoms are interpreted as related to vital substances, internal organs and channels and collaterals, thus constituting the pathological patterns that indicate the state of health of that individual at a given time. Objective: To investigate the interrater agreement and diagnostic performance for pattern differentiation and acupoint prescription under correct diagnosis and misdiagnosis. Methods: This single-blinded, observational cross sectional study compared the interrater agreement of Acupuncture experts for pattern differentiation and acupoints prescription between two groups of human patients simulated with different diagnostic outcomes. A dataset with 509 manifestations of zangfu patterns was organized. Human patients were simulated and separated into groups (n=30 each) according to the diagnostic outcome determined by a computational model: correct and incorrect. A hardcopy questionnaire with 90 questions representing each simulated patient was delivered face-to-face to 6 TCM experts (4 males, 4-year minimal clinic experience) for indicating a single pattern (among 73) and up to 8 acupoints (among 378). Results: Interrater agreement was higher for pattern differentiation than for acupuncture prescription. Interrater agreement on pattern differentiation was slight for both groups with correct diagnosis (Light’s kappa=0.167 [0.108; 0.254]) and incorrect diagnosis outcome (Light’s kappa=0.190 [0.120; 0.286]). Interrater agreement on acupuncture prescription was also slight for both groups of correct (iota=0.029 [0.015; 0.057]) and incorrect diagnosis (iota=0.040 [0.023; 0.058], p=0.075). Analysis of diagnostic performance of raters yielded accuracy=60.9%, sensitivity=21.7%, and specificity=100%. An overall improvement in the interrater agreement and diagnostic accuracy was observed when the data were analyzed using codes for the internal systems instead of the pattern’s labels. Conclusion: Interrater agreement was slight for differentiating zangfu patterns and prescribing acupoints, regardless of whether the diagnosis is accurate or not. Interrater agreement is better for differentiating combinations but yet slight for acupuncture prescription based on the selected internal organs. Diagnostic accuracy of TCM raters for differentiating zangfu patterns was not better than chance for most raters, although it was better for differentiating the affected internal organs in particular when an automated method failed to provide the correct diagnosis.