Can We Share Multiple Choice Questions across Borders? Validation of the Dutch Knowledge Assessment in Family Medicine in Flanders

Background. One of the methods to test knowledge of Family Medicine trainees is a written exam composed of multiple choice questions. Creating high-quality multiple choice questions requires a lot of experience, knowledge, and time. This study explores the opportunity to run the Dutch knowledge assessment in Flanders as well, the use of this test for formative purposes. Methods. The study test was performed in a Flemish sample of postgraduate Family Medicine (FM) trainees and FM trainers. The Dutch test, adjusted to the Flemish context, was analyzed according to the classical test theory: difficulty factor and discriminating power of the items and reliability of the test. Results. 82 of the 154 items well divided the group into two equal parts of correct and incorrect responders. The distribution of the discrimination index, of the items with an acceptable difficulty factor, was [？0.012–0.530]. The item-test-correlation shows that 52 items do not fit, and 87 items need revision in varying degrees. The test reliability was 0.917. Conclusion. The test was highly reliable, but many MC questions appeared to be too easy and poorly discriminative. Therefore, we question the test validity and recommend reconsideration of the items based on difficulty before it is applied and used as a mandatory formative test. 1. Introduction To test the knowledge of medical students, a set of evaluation methods are used: oral exam, Objective Structured Clinical Examination (OSCE), written exam, and so forth. The written exam can differ according to the type of questions used: the most commonly used tests consist of open-ended or multiple choice questions. Multiple choice question exams (MCQ) have been proved to be an adequate way to evaluate knowledge on different educational aspects in the medical curriculum [1, 2]: purely theoretically, knowledge of facts, knowledge of adaptation, and clear understanding of mechanisms [3]. There are two major types of MCQ: one in which the responders need to mark the answers that are appropriate (type true/false). The other type consists of questions in which the examinee has to select one single best answer [2, 4]. The most commonly used MCQ exams consist of questions with only one correct alternative, questions where the beginning of the sentence needs to be completed with one of the alternatives (both type 1) or questions where one alternative presents as the best answer (type 2) [5]. Twenty-five years ago the true/false (MTF) questions were widely used in summative examination. Since then many doubts on both concept and content have