Application of a population-based severity scoring system to individual patients results in frequent misclassification

Fifty-six researchers with explicit training in data abstraction and completion of the AP2 score received scenarios consisting of composites of real patient histories. Descriptive statistics were calculated for each scenario. The standard deviations were calculated compared with an adjudicated score. Intraclass correlations for inter-observer reliability were performed using Shrout-Fleiss methodology. Theoretical distribution curves were calculated for a broad range of AP2 scores using standard deviations of 6, 9 and 12. For each curve, the misclassification rate was determined using an AP2 score cut-off of ≥25. The percentage of misclassifications for each true AP2 score was then applied to the corresponding AP2 score obtained from the PROGRESS severe sepsis registry.The error rate for the total AP2 score was 86% (individual variables were in the range 10% to 87%). Intraclass correlation for the inter-observer reliability was 0.51. Of the patients from the PROGRESS registry. 50% had AP2 scores in the range 17 to 28. Within this interquartile range, 70% to 85% of all misclassified patients would reside.It is more likely that an individual patient will be scored incorrectly than correctly. The data obtained from the scenarios indicated that as the true AP2 score approached an arbitrary cut-off point of 25, the observed misclassification rate increased. Integrating our study of AP2 score errors with the published literature leads us to conclude that the AP2 is an inappropriate sole tool for resource allocation decisions for individual patients.The Acute Physiology and Chronic Health Evaluation II (APACHE II) scoring system was originally developed as a tool for comparing the outcomes of acute disease in critically ill patients across multiple intensive care units in a therapy-independent fashion [1]. Although relatively few critical care units have adopted this system or its successor, APACHE III, for this purpose, APACHE II has found widespread application in clinica