The rating reliability calculator

The program is available public use on our server and the source code is freely available under GNU General Public License. The utility is written in PHP, a common open source imbedded scripting language. The rating data can be entered in a convenient format on the user's personal computer that the program will upload to the server for calculating the reliability and other statistics describing the ratings.When the program is run it displays the reliability, number of subject rated, harmonic mean number of judges rating each subject, the mean and standard deviation of the averaged ratings per subject. The program also displays the mean, standard deviation and number of ratings for each subject rated. Additionally the program will estimate the reliability of an average of a number of ratings for each subject via the Spearman-Brown prophecy formula.This simple web-based program provides a convenient means of estimating the reliability of rating data without the need to conduct special studies in order to provide complete rating data. I would welcome other researchers revising and enhancing the program.Rating scales form an important means of gathering evaluation data in medical education, health services research and quality assurance. Since ratings are often used in making high stakes decisions, assessing the reliability of ratings scales can be critically important. A variety of techniques have been developed for estimating the reliability of ratings. When ratings are in the form of numerical scales, reliability is most commonly assessed using techniques based on classical test theory [1] or more recently, an extension of it, generalizability theory [2].Generalizability theory, which is based on the analysis of sources of variance in the ratings, provides a powerful tool not only for estimating the reliability of ratings but also for developing efficient designs for obtaining reliable ratings. Unfortunately generalizability studies require what is commonly termed a