Say, What Is on Your Mind? Surgeons’ Evaluations of Realism and Usability of a Virtual Reality Vertebroplasty Simulator

Background. Virtual reality (VR)–based simulations offer rich opportunities for surgical skill training and assessment of surgical novices and experts. A structured evaluation and validation process of such training and assessment tools is necessary for effective surgical learning environments. Objective. To develop and apply a classification system of surgeon-reported experience during operation of a VR vertebroplasty simulator. Methods. A group of orthopedic, trauma surgeons and neurosurgeons (n = 13) with various levels of expertise performed on a VR vertebroplasty simulator. We established a mixed-methods design using think-aloud protocols, senior surgical expert evaluations, performance metrics, and a post-simulation questionnaire. Verbal content was systematically analyzed using structured qualitative content analysis. We established a category system for classification of surgeons’ verbal evaluations during the simulation. Furthermore, we evaluated intraoperative performance metrics and explored potential associations with surgeons’ characteristics and simulator evaluation. Results. Overall, 244 comments on realism and usability of the vertebroplasty simulator were collected. This included positive and negative remarks, questions, and specific suggestions for improvement. Further findings included surgeons’ approval of the realism and usability of the simulator and the observation that the haptic feedback of the VR patient’s anatomy requires further improvement. Surgeon-reported evaluations were not associated with performance decrements. Discussion. This study is the first to apply think-aloud protocols for evaluation of a surgical VR-based simulator. A novel classification approach is introduced that can be used to classify surgeons’ verbalized experiences during simulator use. Our lessons learned may be valuable for future research with similar methodological approach