The Effect of Response Order on Item difficulty, Item Discrimination, Distractor Functioning, and Guessing: A Comparison of Listening, Reading, and Grammar Items

Articles in Press

Document Type : Research Paper

Authors

, Department of Foreign Languages and Linguistics, School of Literature and Humanities, Shiraz University, Shiraz, Iran

10.22111/ijals.2026.51273.2502

Abstract

As a means of curbing academic dishonesty in high-stakes multiple-choice examinations, various versions of a given test are generated by manipulating the order of responses, thereby impeding students’ ability to cheat. The current quantitative study employed a between-subjects experimental design to examine the impact of response order on item difficulty, item discrimination, distractor functioning, and guessing across different language skills and components (listening comprehension, grammar, and reading comprehension). An adapted retired version of a TOEFL-PBT test was used to generatefour different versions of the test by changing the response order. Participants were 72 EFL learners who voluntarily took part in the study. They randomly received one of the four test versions. The results indicated significant effects of response order on item difficulty (for 10% of the items), mal-functioning and non-functioning options. However, the effect was not significant for item discrimination, well-functioning options, and guessing. Some traces, though non-significant, were also found for the primacy effect. Finally, the effect of response order did not significantly change across language skills. The results imply that the scores coming from the same test with different response orders are not necessarily comparable, thereby compromising the validity of the decisions made and the test consequences.

Keywords

Iranian Journal of Applied Language Studies

Articles in Press, Accepted Manuscript
Available Online from 22 September 2026

Files

Share

How to cite

Statistics