Psychometric Properties of 3-, 4-, and 5-Option Item Tests: Do Test Takers’ Personality Traits Make a Difference?

Document Type: Research Paper

Authors

Imam Khomeini International University, Qazvin, Iran

Abstract

Prior research has yielded mixed results regarding what contributes psychometrically sound multiple-choice (MC) items. The purpose of the present study was, therefore, twofold: (a) to compare 3-, 4-, and 5-option multiple-choice (MC) tests in terms of psychometric characteristics, and (b) to investigate the relationships between three MC tests and five personality traits. To that end, 150 students were asked to answer three stem equivalent MC item tests. A Big Five Inventory was used to find students’ personality traits. Moreover, an attitude questionnaire was utilized to seek students’ opinions of these three MC tests. The results of one-way repeated measures ANOVA revealed statistically significant differences for item difficulty, while no statistically significant differences were found for item discrimination and reliability across three MC tests. The results of the Pearson correlation showed no correlation between personality traits and three different versions of MC tests. The results of the attitude questionnaire indicated mixed views towards MC tests. The findings of this study suggest that test developers consider statistical, affective, and contextual factors in order to develop different formats of MC tests.

Keywords


Al-Rukban, M. O. (2006). Guidelines for the construction of multiple choice questions tests. Journal of Family & Community Medicine, 13(3), 125-133.
Asmus, E. P. (1981). The effect of altering the number of choices per item on test statistics: Is there better than five?. Bulletin of the Council for Research in Music Education, 65(1), 1-15.
Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford university press.
Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge: Cambridge University Press.
Baghaei, P., & Amrahi, N. (2011). The effects of the number of options on the psychometric characteristics of multiple-choice items. Psychological Test and Assessment Modeling, 53(2), 192-211.
Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. McGraw-Hill College.
Budescu, D. V., & Nevo, B. (1985). Optimal number of options: An investigation of the assumption of proportionality. Journal of Educational Measurement, 22(3), 138-196.
Crehan, K., Haladyna, T. M., & Brewer, B. W. (1993). Use of an inclusive option and the optimal number of options for multiple-choice items. Educational and Psychological Measurement, 53(1), 241-247.
Creswell, J. W., & Clark, V. L. P. (2017). Designing and conducting mixed methods research. California: Sage Publications, Inc.
Currie, M., & Chiramanee, T. (2010). The effect of the multiple-choice item format on the measurement of knowledge of language structure. Language Testing, 27(4), 471-491.
Dawson, J. R. (2015). Comparative analysis on personality traits and motivation on the international student’s academic performance in universities in Taiwan. International Journal of Humanities and Management Sciences, 4(3), 209-216.
Delgado, A. R., & Prieto, G. (1998). Further evidence favoring three-option items in multiple-choice tests. European Journal of Psychological Assessment, 14(3), 197.
Dornyei, Z. (2005). The psychology of the language learner: Individual differences in second language acquisition. London: Lawrence Erlbaum Associates. 
Dornyei, Z., & Ryan, S. (2015). The psychology of the language learner revisited. New York: Routledge.
Ehrman, M. E., & Oxford, R. (1995). Cognition plus: Correlates of language learning success. The Modern Language Journal, 79(1), 67-89. 
Esfandiari, R., & Radfar, S. (2017). An investigation into the relationship between personality traits and Iranian EFL learners’ performance on C-test. Teaching English Language, 11(1), 167-188.
Farhady, H., & Shakery, S. (2000). Number of options and economy of multiple-choice tests. Roshd Foreign Language Teaching Journal, 14(1), 57.
Farhady, H., Ja’farpur, A. & Birjandi, P. (1994). Testing language skills from theory to practice. Tehran: SAMT.
Fulcher, G. (210). Practical language testing. London: Routlege.
Green, K., Sax, G., & Michael, W. (1982). Validity and reliability of tests having differing numbers of options for students of differing level of ability. Educational and Psychological Measurement, 42(1), 239-245.
Haladyna, T. M. (2004). Developing and validating multiple-choice test items. New York: Lawrence Erlbaum Associates.
Haladyna, T. M., & Downing, S. M. (1989). A taxonomy of multiple-choice-item writing rules. Applied Measurement in Education, 2(1), 37-50. doi: 10.1207/s15324818ame0201-3
Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item?. Educational and Psychological Measurement, 53(4), 999-1010.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–334.
Hansen, J. D., & Dexter, L. (1997). Quality multiple-choice test questions: item-writing guidelines and an analysis of auditing test banks. Journal of Education for Business, 73(2), 94-97.
Heaton, J. B. (1988). Writing English language test: Longman handbooks for language teachers. New York: Longman. 
Hogben, D. (2015). The reliability, discrimination and difficulty of word-knowledge tests employing multiple-choice items containing three, four, or five alternatives. The Australian Journal of Education, 17(1), 63-68.
Jensen, M. (2015). Personality traits, learning and academic achievements. Journal of Education and Learning, 4(4), 91-118. doi:10.5539/jel.v4n4p91 
John, O. P., & Srivastava, S. (1999). The Big-Five trait taxonomy: History, measurement, and theoretical perspectives. Handbook of personality. Theory and Research, 2(1999), 102-138. 
John, O. P., Robins, R. W., & Pervin. L. A. (Eds.). (2008). Handbook of personality: Theory and research. New York: Guilford Press.
Kilgour, J. M., & Tayyaba, S. (2016). An investigation into the optimal number of distractors in single-best answer exams. Advances in Health Sciences Education, 21(3), 571-585.
Lee, H., & Winke, P. (2012). The differences among three-, four-, five-option-item format in the context of a high-stakes English-language listening test. Language Testing, 30(1), 99-123. doi: 10.1177/0265532212451235
Mehrens, W. A., & Lehman, I. J. (1991). Measurement and evaluation in education and psychology. Holt, Rinehart and Winston. Inc.: Orlando, FL.
Nejati, R., & Moradi, M. (2015). Utility of complex alternatives in multiple-choice items: The case of all of the above. Research in Applied Linguistics, 6(1), 87-97.
Ng, A., & Chan, A. (2009). Different methods of multiple-choice test, implications and design for further research. Proceedings of the International MultiConference of Engineers and Computer Scientists, 2(1), 1-6.
Nwadinibgwe, P. I., & Naibi, L. (2013). The number of options in a multiple-choice test item and the psychometric characteristics. Journal of Education and Practice, 28(4), 189-196.
Nwadinibgwe, P. I., & Naibi, L. (2013). The number of options in a multiple-choice test item and the psychometric characteristics. Journal of Education and Practice, 28(4), 189-196.
Osterlind, S. J. (2002). Constructing test items multiple-choice, constructed-response, performance, and other formats. Dordrecht: Springer.
Osterlind, S. J. (2002). Constructing test items multiple-choice, constructed-response, performance, and other formats. Dordrecht: Springer.
Owen, S. V., & Froman, R. D. (1987). What’s wrong with three-option multiple-choice items?. Educational and Psychological Measurement, 47(2), 513-522.
Ramos, R. A., & Stern, J. (1973). Item behavior associated with changes in the number of alternatives in multiple-choice items. Journal of Educational Measurement, 10(4), 305-310.
Razavipour, K. (2013). Assessing assessment literacy: Insights from a high-stakes test. Research in Applied Linguistics, 4(1), 111-131.
Richards, J. C., & Schmidt, R. (2002). Longman dictionary of language teaching and applied linguistics. London: Pearson Education Limited.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3-13.
Rogers, W. T., & Harley, D. (1999). An empirical comparison of three-, and four-choice items and tests: Susceptibility to testwiseness and internal consistency reliability. Educational and Psychological Measurement, 59(2), 234-247.
Sadeghi, K., & Akhavan Masoumi, G. (2017). Does number of options in multiple choice tests affect item facility and discrimination? An examination of test-taker preferences. Journal of English Language Teaching and Learning, 9(19), 123-143.
Schneid, S. D., Armour, C., Park, S. Y., Yudkowsky, R., & Bordage, G. (2014). Reducing the number of options on multiple-choice questions: Response time, psychometrics and standard setting. Medical Education, 48(10), 1020-1027.
Shizuka, T., Takeuchi, O., Yashima, T., & Yoshizawa, K. (2006). A comparison of three-and four-option English tests for university entrance selection purposes in Japan. Foreign Language Education and Research, 23(1), 35-57. doi: 10.1191/0265532206lt319oa
Sidick, J. T., Barrett, G. V., & Doverspike, D. (1994). Three-alternative multiple choice tests: An attractive option. Personnel Psychology, 47(4), 829-835.
Swanson, D. B., Holtzman, K. Z., Allbee, K., & Clauser, B. E. (2006). Psychometric characteristics and response times for content-parallel extended-matching and one-best-answer items in relation to number of options. Academic Medicine, 81(10), S52-S55.
Tarrant, M., & Ware, J. (2010). A comparison of the psychometric properties of three-and four-option multiple-choice questions in nursing assessments. Nurse Education Today, 30(6), 539-543.
Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and non-functioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 9(1), 1-8.
Thanyapa, I., & Currie, M. (2014). The number of options in multiple-choice items in language tests: does it make any difference? Evidence from Thailand. Language Testing in Asia, 8(4), 1-21.
Trevisan, M. S., Sax, G., & Michael, W. B. (1994). Estimating the optimum number of options per item using an incremental option paradigm. Educational and Psychological Measurement, 54(1), 86-91.
Vegada, B., Shukla, A., Khilnani, A., Charan, J., & Desai, C. (2016). Comparison between three option, four option and five option multiple choice question tests for quality parameters: A randomized study. Indian Journal of Pharmacology, 48(5), 571-575.