全部 标题 作者
关键词 摘要


Does Internet-Based Survey Have More Stable and Unbiased Results than Paper-and-Pencil Survey?

DOI: 10.4236/jss.2017.51006, PP. 69-86

Keywords: Internet-Based Survey, Paper-and-Pencil Survey, Measurement Reliability, Validity, Equivalence

Full-Text   Cite this paper   Add to My Lib

Abstract:

This study designed to compare responses from an internet-based survey to those from a paper-and-pencil survey in terms of measurement reliability, validity, and equivalence using homogeneous demographic profiles of the undergraduates studying in Taiwan. Several similarities and differences were found between two types of survey in this study. For examples, contents of the survey items (i.e., internet-related vs. behavior-related) and survey environments significantly influence the distribution of responses. The normal distribution, internal consistency, in addition to construct, and convergent validity for individual construct are quite similar. However, the homological validity evidence was demonstrated through structural equation modeling across two survey modes. Implications and future research are also discussed.

References

[1]  Sax, L.J., Gilmartin, S.K. and Bryant, A.N. (2003) Assessing Response Rates and Nonresponse Bias in Web and Paper Surveys. Research in Higher Education, 44, 409-432.
https://doi.org/10.1023/A:1024232915870
[2]  Campos, J.A.D.B., Zucoloto, M.L., Bonafé, F.S.S., Jordani, P.C. and Maroco, J. (2011) Reliability and Validity of Self-Reported Burnout in College Students: A Cross Randomized Comparison of Paper-and-Pencil vs. Online Administration. Computers in Human Behavior, 27, 1875-1883.
https://doi.org/10.1016/j.chb.2011.04.011
[3]  Beuckelaer, A.D., Lievens, F. and Swinnen, G. (2007) Measurement Equivalence in the Conduct of a Global Organizational Survey across Countries in Six Cultural Regions. Journal of Occupational and Organizational Psychology, 80, 575-600.
https://doi.org/10.1348/096317907X173421
[4]  Smither, J.W., Walker, A.G. and Michael, K.T. (2004) An Examination of the Equivalence of Web-Based versus Paper and Pencil Upward Feedback Ratings: Rater and Rate Level Analysis. Educational and Psychology Measurement, 64, 40-61.
https://doi.org/10.1177/0013164403258429
[5]  Stanton, J.M. and Rogelberg, S.G. (2001) Using Internet/Intranet Web Pages to Collect Organizational Research Data. Organizational Research Methods, 4, 200-217.
https://doi.org/10.1177/109442810143002
[6]  Alwin, D.F. (2007) Margins of Error: A Study of Reliability in Survey Measurement. John Wiley and Sons, Inc., Hoboken, NJ. https://doi.org/10.1002/9780470146316
[7]  Meyerson, P. and Tryon, W.W. (2003) Validating Internet Research: A Test of the Psychometric Equivalence of Internet and In-Person Samples. Behavior Research Methods, Instruments, and Computers, 35, 614-620. https://doi.org/10.3758/BF03195541
[8]  Epstein, J., Klinkenberg, W.D., Wiley, D. and McKinley, L. (2001) Insuring Sample Equivalence across Internet and Paper-and-Pencil Assessments. Computers in Human Behavior, 17, 339-346. https://doi.org/10.1016/S0747-5632(01)00002-4
[9]  Beuckelaer, A.D. and Lievens, F. (2009) Measurement Equivalence of Paper-and-Pencil and Internet Organizational Surveys: A Large Scale Examination in 16 Countries. Applied Psychology, 58, 336-361. https://doi.org/10.1111/j.1464-0597.2008.00350.x
[10]  Wang, Y.S. (2007) Development and Validation of a Mobile Computer Anxiety Scale. British Journal of Educational Technology, 38, 990-1009.
https://doi.org/10.1111/j.1467-8535.2006.00687.x
[11]  Yilmaz, V. and Türküm, A.S. (2008) Factors Affecting Hopelessness Levels of Turkish Pre-Teenagers Attending Primary School: A Structural Equation Model. Social Behavior and Personality, 36, 19-26. https://doi.org/10.2224/sbp.2008.36.1.19
[12]  Thompson, L.F., Surface, E.A., Martin, D.L. and Sanders, M.G. (2003) From Paper to Pixels: Moving Personnel Surveys to the Web. Personnel Psychology, 56, 197-227.
https://doi.org/10.1111/j.1744-6570.2003.tb00149.x
[13]  Dillman, D.A. (2000) Mail and Internet Surveys: The Tailored Design Method. 2nd Edition, John Wiley and Sons, New York.
[14]  Yun, G.W. and Trumbo, C.W. (2000) Comparative Response to a Survey Executed by Post, Email and Web Form. Journal of Computer-Mediated Communication, 6, 1-26.
[15]  Simsek, Z. and Veiga, J.F. (2001) A Primer on Internet Organizational Surveys. Organizational Research Methods, 4, 218-235. https://doi.org/10.1177/109442810143003
[16]  Chang, L. and Krosnick, J.A. (2010) Comparing Oral Interviewing with Self-Admi-nistered Computerized Questionnaires an Experiment. Public Opinion Quarterly, 74, 154-167.
https://doi.org/10.1093/poq/nfp090
[17]  Lewis, I.M., Watson, B.C. and White, K.M. (2009) Internet versus Paper-and-Pencil Survey Methods in Psychological Experiments: Equivalence Testing of Participant Responses to Health-Related Messages. Australian Journal of Psychology, 61, 107-116.
https://doi.org/10.1080/00049530802105865
[18]  Martins, N. (2010) Measurement Model Equivalence in Web- and Paper-Based Surveys. Southern African Business Review, 14, 77-107. http://hdl.handle.net/10500/13576
[19]  Manfreda, K.L., Bosnjak, M., Berzelak, J., Haas, I. and Vehovar, V. (2008) Web Surveys versus Other Survey Modes: A Meta-Analysis Comparing Response Rates. International Journal of Market Research, 50, 79-104.
[20]  Hair, J.F., Black, W.C., Babin, B.J., Anderson, R.E. and Tatham, R.L. (2010) Multivariate Data Analysis. 7th Edition, Prentice Hall, Upper Saddle River.
[21]  Schulenberg, S.W. and Yutrzenka, B.A. (1999) The Equivalence of Computerized and Paper-and-Pencil Psychological Instruments: Implications for Measures of Negative Affect. Behavior Research Methods, Instruments, and Computers, 31, 315-321.
https://doi.org/10.3758/BF03207726
[22]  Drasgow, F. and Kanfer, R. (1985) Equivalence of Psychological Measurement in Heterogeneous Populations. Journal of Applied Psychology, 70, 662-680.
https://doi.org/10.1037/0021-9010.70.4.662
[23]  Drasgow, F. (1984) Scrutinizing Psychological Tests: Measurement Equivalence and Equivalent Relations with External Variables Are the Central Issue. Psychological Bulletin, 95, 134-135.
https://doi.org/10.1037/0033-2909.95.1.134
[24]  Raju, N.S., Laffitte, L.J. and Byrne, B.M. (2002) Measurement Equivalence: A Comparison of Methods Based on Confirmatory Factor Analysis and Item Response Theory. Journal of Applied Psychology, 87, 517-529. https://doi.org/10.1037/0021-9010.87.3.517
[25]  Donovan, M.A., Drasgow, F. and Probst, T.M. (2000) Does Computerizing Paper-and-Pencil Job Attitude Scales Make a Difference? New IRT Analyses Offer Insight. Journal of Applied Psychology, 85, 305-313. https://doi.org/10.1037/0021-9010.85.2.305
[26]  Booth-Kewley, S., Edwards, J.E. and Rosenfeld, P. (1992) Impression Management, Social Desirability, and Computer Administration of Attitude Questionnaires: Does the Computer Make the Difference? Journal of Applied Psychology, 77, 562-566.
[27]  Cohen, R.J., Swerdlik, M.E. and Phillips, S.M. (1996) Psychological Testing and Assessment. 3th Edition, Mayfield, Mountain View.
[28]  Cole, M.S., Bedeian, A.G. and Field, H.S. (2006) The Measurement Equivalence of Web-Based and Paper-and-Pencil Measures of Transformational Leadership: A Multinational Test. Organizational Research Methods, 9, 339-368. https://doi.org/10.1177/1094428106287434
[29]  Church, A.H. (2001) Is There a Method to Our Madness? The Impact of Data-Collection Methodology on Organizational Survey Results. Personnel Psychology, 54, 937-969.
https://doi.org/10.1111/j.1744-6570.2001.tb00238.x
[30]  Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, J, Berck, J. and Messer, B.L. (2009) Response Rate and Measurement Differences in Mixed Mode Surveys Using Mail, Telephone, Interactive Voice (IVR) and the Internet. Social Science Research, 38, 1-18.
https://doi.org/10.1016/j.ssresearch.2008.03.007
[31]  Steenkamp, J.B.E.M. and Baumgartner, H. (1998) Assessing Measurement Invariance in Cross-National Consumer Research. Journal of Consumer Research, 25, 78-107.
https://doi.org/10.1086/209528
[32]  Vandenberg, R.J. and Lance, C.E. (2000) A Review and Synthesis of the Measurement Invariance Literature: Suggestions, Practices, and Recommendations for Organizational Research. Organizational Research Methods, 3, 4-70.
https://doi.org/10.1177/109442810031002
[33]  Weijters, B., Schillewaert, N. and Geuens, M. (2008) Assessing Response Styles across Modes of Data Collection. Journal of the Academy of Marketing Science, 36, 409-422.
https://doi.org/10.1007/s11747-007-0077-6
[34]  Cook, T.D. and Campbell, D.T. (1979) Quasi-Experimentation: Design and Analysis Issues for Field Settings. Houghton Mifflin, Boston.
[35]  Tryon, W.W. (2001) Evaluating Statistical Difference, Equivalence, and Indeterminacy Using Inferential Confidence Intervals: An Integrated Alternative Method of Conducting Null Hypothesis Statistical Tests. Psychological Methods, 6, 371-386.
https://doi.org/10.1016/S0378-7206(03)00028-4
[36]  Wang, Y.S. (2003) Assessment of Learner Satisfaction with Asynchronous Electronic Learning Systems. Information and Management, 41, 75-86.
https://doi.org/10.1016/S0378-7206(03)00028-4
[37]  Hui, C., Law, K.S. and Chen, Z.X. (1999) A Structural Equation Model of the Effects of Negative Affectivity, Leader-Member Exchange, and Perceived Job Mobility on In-Role and Extra-Role Performance: A Chinese Case. Organizational Behavior and Human Decision Processes, 77, 3-21. https://doi.org/10.1006/obhd.1998.2812
[38]  Ilies, R., Scott, B.A. and Judge, T.A. (2006) The Interactive Effects of Personal Traits and Experienced States on Intraindividual Patterns of Citizenship Behavior. Academy of Management Journal, 49, 561-575. https://doi.org/10.5465/AMJ.2006.21794672
[39]  Kline, R.B. (2005) Principles and Practice of Structural Equation Modeling. 2nd Edition, The Guilford Press, New York.
[40]  Hu, L.T. and Bentler, P.M. (1999) Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Structural Equation Modeling, 6, 1-55.
https://doi.org/10.1080/10705519909540118
[41]  Cronbach, L.J. and Shavelson, R.J. (2004) My Current Thoughts on Coefficient Alpha and Successor Procedures. Educational and Psychological Measurement, 64, 391-418.
https://doi.org/10.1177/0013164404266386
[42]  Robinson, J.P., Shaver, P.R. and Wrightsman, L.S. (1991) Criteria for Scale Selection and Evaluation. In: Robinson, J.P., Shaver, P.R. and Wrightsman, L.S., Eds., Measures of Personality and Social Psychological Attitudes, Academic Press, San Diego, 1-15.
https://doi.org/10.1016/B978-0-12-590241-0.50005-8
[43]  Field, A. (2005) Discovering Statistics Using SPSS. 2nd Edition, Sage, London.
[44]  Vispoel, W.P., Boo, J. and Bleiler, T. (2001) Computerized and Paper-and-Pencil Versions of the Rosenberg Self-Esteem Scale: A Comparison of Psychometric Features and Respondent Preferences. Educational and Psychological Measurement, 61, 461-474.
https://doi.org/10.1177/00131640121971329
[45]  Ployhart, R.E., Weekley, J.A., Holtz, B.C. and Kemp, C. (2003) Web-Based and Paper-and-Pencil Testing of Applicants in a Proctored Setting: Are Personality, Biodata, and Situational Judgment Tests Comparable? Personnel Psychology, 56, 733-752.
[46]  Vandenberg, R.J. (2002) Toward a Further Understanding of and Improvement in Measurement Invariance Methods and Procedures. Organizational Research Methods, 5, 139-158.
https://doi.org/10.1177/1094428102005002001

Full-Text

comments powered by Disqus