Background and Purpose. In this paper, we present a protocol for advanced psychometric assessments of surveys based on the Standards for Educational and Psychological Testing. We use the Alberta Context Tool (ACT) as an exemplar survey to which this protocol can be applied. Methods. Data mapping, acceptability, reliability, and validity are addressed. Acceptability is assessed with missing data frequencies and the time required to complete the survey. Reliability is assessed with internal consistency coefficients and information functions. A unitary approach to validity consisting of accumulating evidence based on instrument content, response processes, internal structure, and relations to other variables is taken. We also address assessing performance of survey data when aggregated to higher levels (e.g., nursing unit). Discussion. In this paper we present a protocol for advanced psychometric assessment of survey data using the Alberta Context Tool (ACT) as an exemplar survey; application of the protocol to the ACT survey is underway. Psychometric assessment of any survey is essential to obtaining reliable and valid research findings. This protocol can be adapted for use with any nursing survey. 1. The Alberta Context Tool Organizational context is “…the environment or setting in which people receive healthcare services, or in the context of getting research evidence into practice, the environment or setting in which the proposed change is to be implemented” ([1], page 299). Context is believed to influence the successful implementation of research evidence by nurses in healthcare settings internationally. However, there is little empirical evidence to support this claim. One reason for this is the absence of a robust measure of organizational context in healthcare. The Alberta Context Tool (ACT) was developed in 2006 to address this gap. Underpinned by the Promoting Action on Research Implementation in Health Services (PARiHS) framework [1, 2] and related literature [3, 4], the ACT was constructed to measure healthcare providers’ and care managers’ perceptions of modifiable dimensions of organizational context; their responses can then be aggregated to provide nursing unit and/or organizational (e.g., hospital or nursing home or home care office) estimates of context. Three principles informed the development of the ACT: use of the PARiHS framework and related literature to identify a comprehensive set of contextual concepts, brevity—it could be completed in 20 minutes or less, and a focus on modifiable (and therefore researchable) elements of context
References
[1]
J. Rycroft-Malone, “The PARIHS framework—a framework for guiding the implementation of evidence-based practice,” Journal of Nursing Care Quality, vol. 19, no. 4, pp. 297–304, 2004.
[2]
A. Kitson, G. Harvey, and B. McCormack, “Enabling the implementation of evidence based practice: a conceptual framework,” Quality and Safety in Health Care, vol. 7, no. 3, pp. 149–158, 1998.
[3]
M. Fleuren, K. Wiefferink, and T. Paulussen, “Determinants of innovation within health care organizations. Literature review and Delphi study,” International Journal for Quality in Health Care, vol. 16, no. 2, pp. 107–123, 2004.
[4]
T. Greenhalgh, G. Robert, F. Macfarlane, P. Bate, and O. Kyriakidou, “Diffusion of innovations in service organizations: systematic review and recommendations,” The Milbank Quarterly, vol. 82, no. 4, pp. 581–629, 2004.
[5]
C. A. Estabrooks, J. E. Squires, G. G. Cummings, J. M. Birdsell, and P. G. Norton, “Development and assessment of the Alberta Context Tool,” BMC Health Services Research, vol. 9, article 234, 2009.
[6]
C. A. Estabrooks, J. E. Squires, A. M. Adachi, L. Kong, and P. G. Norton, “Utilization of health research in acute care settings in Alberta,” Tech. Rep., Faculty of Nursing, University of Alberta, Edmonton, Canada, 2008.
[7]
J. E. Squires, C. A. Estabrooks, L. Kong, and S. Brooker, “Examining the role of context in Alzheimer Care centers: a pilot study,” Tech. Rep. 0804-TR, Faculty of Nursing, Edmonton, Canada, 2009.
[8]
A. M. Hutchinson, L. Kong, A. M. Adachi, C. A. Estabrooks, and B. Stevens, “Context and research use in the care of children: a pilot study,” Tech. Rep., Faculty of Nursing, University of Alberta, Edmonton, Canada, 2008.
[9]
C. A. Estabrooks, J. E. Squires, L. A. Hayduk, G. G. Cummings, and P. G. Norton, “Advancing the argument for validity of the Alberta context tool with healthcare aides in residential long-term care,” BMC Medical Research Methodology, vol. 11, article 107, 2011.
[10]
D. Streiner and G. Norman, Measurement Scales: A Practical Guide to their Development and Use, Oxford University Press, Oxford, UK, 4th edition, 2008.
[11]
S. Messick, “Validity,” in Educational Measurement, R. L. Linn, Ed., American Council on Education, New York, NY, USA, 3rd edition, 1989.
[12]
S. Messick, “Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning,” American Psychologist, vol. 50, no. 9, pp. 741–749, 1995.
[13]
S. Messick, “Validity and washback in language testing,” Language Testing, vol. 13, no. 3, pp. 241–256, 1996.
[14]
M. T. Kane, “An argument-based approach to validity,” Psychological Bulletin, vol. 112, no. 3, pp. 527–535, 1992.
[15]
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, Standards For Educational and Psychological Testing, American Educational Research Association, Washington, DC, USA, 1999.
[16]
J. Nunnally and I. Bernstein, Psychometric Theory, McGraw-Hill, New York, NY, USA, 3rd edition, 1994.
[17]
C. F. Waltz, O. Strickland, and E. Lenz, Measurement in Nursing and Health Research, Springer, New York, NY, USA, 2005.
[18]
B. J. Kalisch, H. Lee, and E. Salas, “The development and testing of the nursing teamwork survey,” Nursing Research, vol. 59, no. 1, pp. 42–50, 2010.
[19]
R. M. Guion, “On Trinitarian doctrines of validity,” Professional Psychology, vol. 11, no. 3, pp. 385–398, 1980.
[20]
J. Nunnally and I. Bernstein, Psychometric Theory, McGraw-Hill, New York, NY, USA, 1994.
[21]
W. J. Van der Linden and R. K. Hambleton, Handbook of Modern Item Response Theory, Springer, New York, NY, USA, 1997.
[22]
N. E. Betz, “Test construction,” in The Psychology Research Handbook: A Guide for Graduate Students and Research Assistants, J. T. Leong FTLAustin, Ed., pp. 239–250, Sage, Thousand Oaks, Calif, USA, 2000.
[23]
P. Barrett, “Structural equation modelling: adjudging model fit,” Personality and Individual Differences, vol. 42, no. 5, pp. 815–824, 2007.
[24]
L. Hayduk, G. Cummings, K. Boadu, H. Pazderka-Robinson, and S. Boulianne, “Testing! testing! one, two, three—testing the theory in structural equation models!,” Personality and Individual Differences, vol. 42, no. 5, pp. 841–850, 2007.
[25]
L. T. Hu and P. M. Bentler, “Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives,” Structural Equation Modeling, vol. 6, no. 1, pp. 1–55, 1999.
[26]
B. M. Byrne, Structural Equation Modeling, Sage, Thousand Oaks, Calif, USA, 1994.
[27]
W. H. Glick :, “Conceptualizing and measuring organizational and psychological climate: pitfalls in multilevel research,” Academy of Management Review, vol. 10, pp. 601–616, 1985.
[28]
R. Rosenthal and R. L. Rosnow, Essentials of Behavioural Research: Methods and Data Analysis, McGraw Hill, New York, NY, USA, 2nd edition, 1991.
[29]
P. D. Bliese, “Within-group agreement, non-independence, and reliability: implications for data aggregation and analysis,” in Multilevel Theory, Research, and Methods in Organizations: Foundations, Extensions, and New Directions, K. J. Klein and S. W. J. Kozlowski, Eds., pp. 349–381, Jossey-Bass, San Francisco, Calif, USA, 2000.
[30]
G. Keppel, Design and Analysis: A Researcher'S Handbook, Prentice-Hall, Englewood Cliffs, NJ, USA, 1991.
