Automated Comparison of Pathology Reports for On-the-job Assessment of Residents

The clinical competency of residents at teaching hospitals is always under scrutiny. Ideally, assessment should reflect competency on-the-job, under realistic circumstances, and include evaluating their medical reports. Currently, the assessment is done manually by the attending physicians, which adds to the cognitive load. In this study, we developed an automated system for assessing medical resident’s pathology reports. Our system used natural language processing (NLP) techniques to identify different lexical and semantic similarity scores at sentence level as well as chunk level. We then used supervised learning to classify the reports into three categories– No Change (NC), Minor Changes (MiC), and major changes (MaC), reflecting how much the attending physician’s report differs from that of the resident. Our system was able to classify the reports with an accuracy of 73.6%. Although moderately successful, our work shows the potential and future of automated assessment systems in the biomedical domain