This study attempted to determine the effectiveness of the number of extra negative items in identifying insufficient effort responses in Attitudes toward Statistics (SATS-36). The (SATS-36), which consists of 36 5-point Likert Scale items, was used to actually achieve the goal of this study. Furthermore, the researchers developed three forms, each with a different set of extra negative items (2, 4, and 6). The three forms were distributed to a sample of (750) students at Yarmouk University. The results revealed that form 1, which enclosed 6 extra negative items, had the lowest detected rate of insufficient effort responses (IERs) (7.20%), while form 3, which contained 2 extra negative items, had the highest detection rate (15.6%). The detection rate was discovered respectively among the Lie Detection Scale, Mahalanobois, lpz and the method of extra negative items. The highest detection rate was found in form 3 with two extra negative items, and data reliability decreased after the insufficient effort responses were removed (IERs). Furthermore, the results showed that the maximum changes in model-data fit indices after data filtering were in form 3, which contained two extra negative items. Moreover, the results indicate that the reliability of data after filtering those with insufficient effort responses (IERs) is reduced.
References
[1]
Kountur, R. (2016) Detecting Careless Responses to Self-Reported Questionnaires. Eurasian Journal of Educational Research, 64, 307-318. https://doi.org/10.14689/ejer.2016.64.17
[2]
Meijer, R.R. and Nering, M.L. (1997) Trait Level Estimation for Nonfitting Response Vectors. Applied Psychological Measurement, 21, 321-336. https://doi.org/10.1177/01466216970214003
[3]
Huang, J.L., Curran, P.G., Keeney, J., Poposki, E.M. and DeShon, R.P. (2012) Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business and Psychology, 27, 99-114. https://doi.org/10.1007/s10869-011-9231-8
[4]
Meade, A.W. and Craig, S.B. (2012) Identifying Careless Responses in Survey Data. Psychological Methods, 17, 437-455. https://doi.org/10.1037/a0028085
[5]
McKay, A.S. (2014) Improving Data Quality with Four Short Sentences: How an Honor Code Can Make the Difference during Data Collection. M.A. Thesis, California State University, San Bernardino.
[6]
Thompson, A.H. (1975) Random Responding and the Questionnaire Measurement of Psychoticism. Social Behavior and Personality: An International Journal, 3, 111-115. https://doi.org/10.2224/sbp.1975.3.2.111
[7]
Zijlstra, W.P., van der Ark, L.A. and Sijtsma, K. (2011) Outliers in Questionnaire Data: Can They Be Detected and Should They Be Removed? Journal of Educational and Behavioral Statistics, 36, 186-212. https://doi.org/10.3102/1076998610366263
[8]
Rios, J.A., Guo, H., Mao, L. and Liu, O.L. (2017) Evaluating the Impact of Careless Responding on Aggregated-Scores: To Filter Unmotivated Examinees or Not? International Journal of Testing, 17, 74-104. https://doi.org/10.1080/15305058.2016.1231193
[9]
Eysenck, H.J. and Eysenck, S.G.B. (1965) The Eysenck Personality Inventory. British Journal of Educational Studies, 14, Article No. 140.
[10]
Tendeiro, J.N., Meijer, R.R., Niessen, A.S.M., et al. (2016) PerFit: An R Package for Person-Fit Analysis in IRT. Journal of Statistical Software, 74, 1-27. https://doi.org/10.18637/jss.v074.i05
[11]
Snijders, T.A.B. (2001) Asymptotic Null Distribution of Person Fit Statistics with Estimated Person Parameter. Psychometrika, 66, 331-342. https://doi.org/10.1007/BF02294437
[12]
Levine, M.V. and Rubin, D.B. (1979) Measuring the Appropriateness of Multiple-Choice Test Scores. Journal of Educational and Behavioral Statistics, 4, 269-290. https://doi.org/10.3102/10769986004004269
[13]
Conijn, J.M., Emons, W.H.M. and Sijtsma, K. (2014) Statistic lz-Based Person-Fit Methods for Noncognitive Multiscale Measures. Applied Psychological Measurement, 38, 122-136. https://doi.org/10.1177/0146621613497568
[14]
Swain, S.D., Weathers, D. and Niedrich, R.W. (2008) Assessing Three Sources of Misresponse to Reversed Likert Items. Journal of Marketing Research, 45, 116-131. https://doi.org/10.1509/jmkr.45.1.116
[15]
Schriesheim, C.A. and Hill, K.D. (1981) Controlling Acquiescence Response Bias by Item Reversals: The Effect on Questionnaire Validity. Educational and Psychological Measurement, 41, 1101-1114. https://doi.org/10.1177/001316448104100420
[16]
Bradley, K.D., Royal, K.D. and Bradley, J.W. (2008) An Investigation of ‘Honesty Check’ Items in Higher Education Course Evaluations. Journal of College Teaching & Learning, 5, 39-48. https://doi.org/10.19030/tlc.v5i8.1240
[17]
Hinz, A., Michalski, D., Schwarz, R. and Herzberg, P.Y. (2007) The Acquiescence Effect in Responding to a Questionnaire. Psycho-Social-Medicine, 4, 1-9.
[18]
Van Sonderen, E., Sanderman, R. and Coyne, J.C. (2013) Ineffectiveness of Reverse Wording of Questionnaire Items: Let’s Learn from Cows in the Rain. PLOS ONE, 8, e68967. https://doi.org/10.1371/journal.pone.0068967
[19]
Józsa, K. and Morgan, G.A. (2017) Reversed Items in Likert Scales: Filtering Out Invalid Responders. Journal of Psychological and Educational Research, 25, 7-25.
[20]
Schwarz, R. and Hinz, A. (2001) Reference Data for the Quality of Life Questionnaire EORTC QLQC30 in the General German Population. European Journal of Cancer, 37, 1345-1351. https://doi.org/10.1016/S0959-8049(00)00447-0
[21]
Baer, R.A., Ballenger, J., Berry, D.T.R. and Wetter, M.W. (1997) Detection of Random Responding on the MMPI-A. Journal of Personality Assessment, 68, 139-151. https://doi.org/10.1207/s15327752jpa6801_11
[22]
Schau, C. (2003) Students’ Attitudes: The “Other” Important Outcome in Statistics Education. 2003 Joint Statistical Meetings-Section on Statistical Education, San Francisco, 3673-3681
[23]
Saraierh, R. (2013) Construct Validity of the Arabic Version of Arabic version of the Measure in Attitudes toward Statistics Scale (SATS-36). Faculty of Education-Ain Shams University, 3, 651-672.
[24]
Al-Sharim, A. (2015) Invariance of Factor Structure to Survey Attitudes toward Statistics (SATS-36) by Administration Time of Scale. The International Journal of Interdisciplinary Educational Studies, 4, 14-31.
[25]
DeSimone, J.A., Harms, P.D. and DeSimone, A.J. (2015) Best Practice Recommendations for Data Screening. Journal of Organizational Behavior, 36, 171-181. https://doi.org/10.1002/job.1962
[26]
Hendrawan, I., Glas, C.A.W. and Meijer, R.R. (2005) The Effect of Person Misfit on Classification Decisions. Applied Psychological Measurement, 29, 26-44. https://doi.org/10.1177/0146621604270902
[27]
Hult, G.T.M., Hair Jr, J.F., Proksch, D., Sarstedt, M., Pinkwart, A. and Ringle, C.M. (2018) Addressing Endogeneity in International Marketing Applications of Partial Least Squares Structural Equation Modeling. Journal of International Marketing, 26, 1-21. https://doi.org/10.1509/jim.17.0151
[28]
Steedle, J.T., Hong, M. and Cheng, Y. (2019) The Effects of Inattentive Responding on Construct Validity Evidence When Measuring Social-Emotional Learning Competencies. Educational Measurement Issues and Practice, 38, 101-111. https://doi.org/10.1111/emip.12256
[29]
Al Quraan, M. (2019) The Effect of Insufficient Effort Responding on the Validity of Student Evaluation of Teaching. Journal of Applied Research in Higher Education, 11, 604-615. https://doi.org/10.1108/JARHE-03-2018-0034
[30]
Cheung, G.W. and Rensvold, R.B. (2009) Structural Equation Modeling: A Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9, 233-255. https://doi.org/10.1207/S15328007SEM0902_5
[31]
Breivik, E. and Olsson, U.H. (2001) Adding Variables to Improve Fit: The Effect of Model Size on Fit Assessment in LISREL. In: Cudeck, R., Du Toit, S., Sorbom, D., Eds., Structural Equation Modeling: Present and Future, Scientific Software International, Lincolnwood, IL, 169-194.
[32]
Steedle, J. (2018) Detecting Inattentive Responding on a Psychosocial Measure of College Readiness. Research Report 2018-5, ACT, Inc., Iowa City.
