Score reports are critical to the valid use of test scores and the interpretability of score reports has been recommended to be included as one type of validity evidence in validation research. In practice, how to support the validity argument for a score report has become the focus of practitioners’ concern. In this paper, the authors first emphasize the significance of score reporting in test development before going on to review what a score report is, what it may possibly contain, validity and validation. Based on previous literature on validity of reports, a four-source framework of validity evidence for a score report is thus proposed. The four sources are 1) content alignment, 2) users’ correct interpretation, 3) users’ appropriate actions and 4) users’ perception. Possible methods of collecting these different kinds of evidence are also suggested.
References
[1]
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME) (1985). Standards for Educational and Psychological Tests and Mannuals. American Educational Research Association.
[2]
American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME) (2014). Standards for Educational and Psychological Tests and Mannuals. American Educational Research Association.
[3]
American Psychological Association (1954). Technical Recommendations for Psychological Test and Diagnostic Techniques. Author.
[4]
Bachman, L. F., & Palmer, A. (2010). Language Assessment in Practice. Oxford University Press.
[5]
Bachman, L. F., & Palmer, A. S. (1996). Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford University Press.
[6]
Bennett, R. E. (2011). Formative Assessment: A Critical Review. Assessment in Education: Principles, Policy & Practice, 18, 5-25.
https://doi.org/10.1080/0969594X.2010.513678
[7]
Bosker, R. J., Branderhorst, E. M., & Visscher, A. J. (2007). Improving the Utilisation of Management Information Systems in Secondary Schools. School Effectiveness and School Improvement, 18, 451-467. https://doi.org/10.1080/09243450701712577
[8]
Carroll, J. B. (1968). The Psychology of Language Testing. In A. Davies (Ed.), Language Testing Symposium: A Psycholinguistic Approach (pp. 46-69). Oxford University Press.
[9]
Downing, S. M., & Haladyna, T. M. (2006). Handbook of Test Development. Erlbaumb.
[10]
Gandara, F., & Rick, F. (2017). Empowering End Users to Evaluate Score Reports: Current Challenges and Proposed Solution. Pensamiento Educativo, Revista de Investigación Educacional Latinoamericana, 54, 1-24. https://doi.org/10.7764/PEL.54.2.2017.11
[11]
Gorney, B. T. (1999). The GRE Score Report: Exploring Alternatives for Presentation of Individual and Aggregate Scores to Institutional Recipients. Unpublished Ph.D. Thesis, University of North California.
[12]
Guion, R. M. (1980). On Trinitarian Conceptions of Validity. Professional Psychology, 11, 385-398. https://doi.org/10.1037/0735-7028.11.3.385
[13]
Hambleton, R. K., & Zenisky, A. L. (2013). Reporting Test Scores in More Meaningful Ways: A Research-Based Approach to Score Report Design. In K. F. Geisinger (Ed.), APA Handbook of Testing and Assessment in Psychology: Vol. 3 Testing and Assessment in School Psychology and Education (pp. 479-494). American Psychological Association. https://doi.org/10.1037/14049-023
[14]
Hattie, J. (2009). Visibly Learning from Reports: The Validity of Score Reports. Online Educational Research Journal. http://www.oerj.org/View?action=viewPDF&paper=6
[15]
Henning, G. (1987). A Guide to Language Testing. Newbury House.
[16]
Impara, J. C., Divine, K. P., Bruce, F. A., Liverman, M. R., & Gay, A. A. (1991). Does Interpretive Test Score Information Help Teachers? Educational Measurement: Issues and Practice, 10, 16-18. https://doi.org/10.1111/j.1745-3992.1991.tb00212.x
[17]
Kane, M. T. (2013). Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement, 50, 1-73. https://doi.org/10.1111/jedm.12000
[18]
Klesch, H. S. (2010). Score Reporting in Teacher Certification Testing: A Review, Design, and Interview/Focus Group Study (p. 251). Open Access Dissertations.
[19]
MacIver, R., Anderson, N. A., & Evers, A. (2014). Validity of Interpretation: A User Validity Perspective beyond the Test Score. International Journal of Selection, 22, 149-164.
https://doi.org/10.1111/ijsa.12065
[20]
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed., pp. 13-103). Macmillan.
[21]
O’Leary, T. M. (2018). Effective Score Reporting: Establishing Evidence-Informed Principles for Outcomes Focused Score Reports. Doctoral Dissertation.
[22]
O’Leary, T. M., Hattie, J. A., & Griffin, P. (2017). Actual Interpretations and Use of Scores as Aspects of Validity. Educational Measurement: Issues and Practice, 36, 16-23.
https://doi.org/10.1111/emip.12141
[23]
Rankin, J. (2016). Standards for Reporting Data to Educators: What Educational Leaders Should Know and Demand. Routledge. https://doi.org/10.4324/9781315623283
[24]
Ryan, J. M. (2006). Practices, Issues, and Trends in Student Test Score Reporting. In S. M. Downing, & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 677-710). Erlbaumb.
[25]
Slater, S., Livingston, S. A., & Silver, M. (2018). Score Reports for Large-Scale Testing Programs. In D. Zapata-Rivera (Ed.), Score Reporting Research and Applications (pp. 174-200). Routledge. https://doi.org/10.4324/9781351136501-8
[26]
Tannenbaum, R. J. (2018). Validity Aspects of Score Reporting. In D. Zapata-Rivera (Ed.), Score Reporting Research and Applications (pp. 33-52). Routledge.
https://doi.org/10.4324/9781351136501-2
[27]
Van der Kleij, F. M., Eggen, T. J. H. M., & Engelen, R. J. H. (2014). Towards Valid Score Reports in the Computer Program LOVS: A Redesign Study. Studies in Educational Evaluation, 43, 24-39. https://doi.org/10.1016/j.stueduc.2014.04.004
[28]
Vanhoof, J., Verhaeghe, G., Verhaeghe, J. P., Valcke, M., & Van Petegem, P. (2011). The Influence of Competences and Support on School Performance Feedback Use. Educational Studies, 37, 141-154. https://doi.org/10.1080/03055698.2010.482771
[29]
Zapata-Rivera, D., & Katz, R. I. (2014). Keeping Your Audience in Mind: Applying Audience Analysis to the Design of Interactive Score Reports. Assessment in Education: Principles, Policy & Practice, 21, 442-463.
https://doi.org/10.1080/0969594X.2014.936357
[30]
Zenisky, A. L., & Hambleton, R. K. (2012). Developing Test Score Reports That Work: The Process and Best Practices for Effective Communication. Educational Measurement: Issues and Practice, 31, 21-26. https://doi.org/10.1111/j.1745-3992.2012.00231.x
[31]
Zenisky, A. L., & Hambleton, R. K. (2015). Good Practices for Score Reporting. In S. Lane, M. R. Raymond, & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 585-602). Routledge.
