The relative impact of persons, items, subtests, and academic background on performance on a language proficiency test

This study exploited generalizability theory to explore the impact of persons, items, subtests, and academic background on the dependability of the scores from a high-stakes language proficiency test, the University of Tehran English Proficiency Test (UTEPT). To this end and following Brown (1999), three questions were posed: 1. What are the distributional characteristics and CTT reliability of UTEPT test scores? 2. What are the relative contributions of persons, items, and subtests to the dependability of scores for each group and for all the groups combined? 3. What are the relative contributions of persons, items, subtests, academic background as well as their various interactions to the dependability of the scores when all groups are combined? To investigate the issues, 5795 examinees from four different academic backgrounds were selected from among all the participants who had taken the test in 2004. The results of the study indicated that the relative contributions of the facets were not stable across all groups, though highly similar. In addition, with academic background added as a facet, there was no significant interaction between items and fields, and the dependability of the scores did not decrease either. This result shows that background knowledge does not lead to bias in the UTEPT. This use of G-theory could be extended profitably to other measuring situations.