Non-native English Speaking Teachers’ Pragmatic Criteria in the Holistic and Analytic Rating of the Agreement Speech Act Productions of Iranian EFL Learners

Document Type : Research Paper

Authors

1 West Tehran Branch, Islamic Azad University,Tehran, Iran

2 Sharif University of Technology, Tehran, Iran

Abstract

Pragmatic rating is considered one of the novel and crucial aspects of second language education which has not been maneuvered upon in the literature. To address this gap, the current study aimed to inspect the matches and mismatches, to explore rating variations, and to assess the rater consistency between the holistic and analytic rating methods of the speech act of agreement in L2 by non-native English teachers. In this regard, 12 discourse completion tests (DCTs) for agreement accompanied by EFL learners’ responses to each situation were rated by 50 non-native English teachers. Initially, they were asked to rate it holistically, and the content analysis of raters’ comments revealed twelve agreement criteria. Grammatical structure was the prominent criterion which suggested that the raters were predominantly concerned with pragmalinguistics. In particular, the results of descriptive statistics demonstrated that there was a degree of divergence in the frequency of the criteria applied. Additionally, the teachers were asked to rate the pragmatic outputs analytically which showed that the raters were more consistent in the analytic phase. Finally, the findings indicated that there was a convergence between the two rating methods. The results of the present study implicated the necessity of rater training with regard to the rubric-based pragmatic rating. This study offers crucial pedagogical implications for syllabus designers, materials developers, language testers, and some suggestions for further research studies.

Keywords

References

Alcon Soler, E., & Martinez Flor, A. (Eds.). (2008). Investigating pragmatics in foreign language learning, teaching and testing. Clevedon: Multilingual Matters.
Alemi, M., Eslami Rasekh, Z., R., & Rezanejad, A. (2014). Rating EFL learners’ interlanguage pragmatic competence by non-native English speaking teachers. Procedia-Social and Behavioral Sciences, 98, 171-174.
Alemi, M., Eslami Rasekh, Z., & Rezanejad, A. (2015). Iranian non-native English speaking teachers’ rating criteria regarding the speech act of compliment: An investigation of teachers’ variables. Journal of Teaching Language Skills, 6(3),21-49.
Alemi, M., & Khanlarzadeh, N. (2016). Pragmatic assessment of request speech act of Iranian         EFL learners by non-native English speaking teachers. Iranian Journal of Language Teaching Research, 4(2), 19-34.
Alemi, M., & Khanlarzadeh, N. (2017). Native and non-native teachers’ pragmatic criteria for rating request speech act: The case of American and Iranian EFL teachers. Applied Research on English Language, 6(1), 67-84.
Alemi, M., & Motamedi, M. (2019). Pragmatic criteria in the holistic and analytic rating of the disagreement speech act of Iranian EFL learners by non-native English speaking teachers. Journal of Teaching Language Skills, 38(1), 1-36.
Alemi, M., & Tajeddin, Z. (2013). Pragmatic rating of L2 refusal: Criteria of native and non-native English teachers. TESL Canada Journal, 30(7),63-81.
Al-Khanaifsawy, A. N. (2016). Investigating Iraqi EFL learners’ use of the speech act of agreement. Adab-alkufa. 27(1), 11-30.
Bachman, L. F., Lynch, B. K., & Mason, M. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12(2), 238-257.
Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge, UK: Cambridge University Press.
Bardovi-Harlig, K. (2001). Evaluating the empirical evidence: Grounds for instruction in Pragmatics?. In K. Rose & G. Kasper (Eds.), Pragmatics in language teaching (pp. 13-32). Cambridge, UK: Cambridge University Press.
Barraja-Rohan, A. M. (2011). Using conversation analysis in the second language classroom to teach interactional competence. Language Teaching Research, 15(4), 479-507.
Birner, B. J. (2012). Introduction to pragmatics. Malden, MA: Wiley-Blackwell.
Blum-Kulka, S., & Olshtain, E. (1984). Requests and apologies: A cross-cultural study of speech act realization patterns (CCSARP). Applied Linguistics, 5(3), 196-213.
Brown, P., & Levinson, S. (1987). Politeness: Some universals in language usage. Cambridge: Cambridge University Press.
Carr, N. T. (2000). A comparison of the effects of analytic and holistic rating scale types in the context of composition tests. Issues in Applied Linguistics, 11(2), 207-229.
Cohen, A. D. (2008). Teaching and assessing L2 pragmatics: What can we expect from learners? Language Teaching, 41(2), 213-235.
Crystal, D. (1997). The Cambridge encyclopedia of language (2nd ed.). Oxford: MIT Press.
Douglas, D., & Smith, J. (1997). Theoretical underpinnings of the Test of Spoken English Revision Project (TOEFL Monograph Series Report No. 9).Princeton, New Jersey: Educational Testing Service.
Eslami-Rasekh, Z. (2005). Raising the pragmatic awareness of language learners. ELT Journal, 59(3), 199-208.
Eslami, A., Jafari, D. S., & Mehregan, M. (2012). How do you react to the breakdown after it happens? Do you complain about it? :  A contrastive study on the complaint behavior in American English and Persian. Procedia-Social and Behavioral Sciences, 47, 34-40.
Fraser, B. (1981). On apologizing. In F. Coulmas (Ed.), Conversational Routine: Explorations in standardized communication situations and pre-patterned speech (pp. 259-271). New York: Mouton.
Fulcher, G. (2003). Testing second language speaking. London: Pearson Education Limited.
Godshalk, F. I., Swineford, F., & Coffman, W.E. (1966). The measurement of writing ability. New York: College Entrance Examination Board.
Hamp-Lyons, L. (1991). Scoring procedures for ESL contexts. In L. Hamp-Lyons (Ed.), Assessing second language writing in academic contexts (pp. 241-276). Norwood, NJ: Ablex.
Holmes, J. (1990). Apologies in New Zealand English. Language in Society, 19(2), 155-199.
Hunter, D. M., Jones, R. M., & Randhawa, B. S. (1996). The use of holistic versus analytic scoring for large-scale assessment of writing. The Canadian Journal of Program Evaluation, 11(2), 61-85.
Jalilifar, A. (2009). Request strategies: Cross-sectional study of Iranian EFL learners and Australian native speakers. English Language Teaching, 2(1), 46-61.
Jianda, L. (2006). Assessing EFL learners’ interlanguage pragmatic knowledge: Implications for testers and learners. Reflections on English Language Teaching 5 (1), 1-22.
Johnson, F. (2006).  Agreement and disagreement: A cross-cultural comparison. BISAL, 1, 41-67.
Ishihara, N., & Cohen, A. D. (2010). Teaching and learning pragmatics: Where language and culture meet. Harlow, UK: Pearson Education.
McNamara, T. F. (1996). Measuring second language performance. London: Longman.
Murphy, B., & Neu, J. (1996). My grade’s too low: The speech act set of complaining. In S. M. Gass, & J. Neu (Eds.), Speech acts across cultures: Challenges to communication in a second language (pp. 191-216). Berlin: Mouton de Gruyter.
Olshtain, S., & Cohen, A. D. (1983). Apology: A speech act set. In N. Wolfson & E. Judd (Eds.), Sociolinguistics and language acquisition (pp. 18-35). Rowley, MA: Newbury House.
Olshtain, E., Blum Kulka, S. (1985). Degree if approximation: Nonnative reactions to native speech act behavior. In S. M. Gass, & C. G. Madden (Eds.), Input in second language acquisition (pp. 303-325). Rowley, MA: Newbury House.
Pomerantz, A. (1984). Agreeing and disagreeing with assessments: Some features of preferred / dispreferred turn shapes. In J. Atkinson & J. M. Heritage (Eds.), Structures in Social Action (pp. 57-101). Cambridge: Cambridge University Press.
Rose, K. R., & Kasper, G. (Eds.). (2001). Pragmatics in language teaching. New York: Cambridge University Press.
Schegloff, E. A. (1996). Confirming allusions: Toward an empirical account of action. American journal of sociology, 102(1), 161-216.
Scollon, R., & Scollon, S.W. (2001). 27 discourse and intercultural communication. In D. Schiffrin, D. Tannen, & H. Hamilton (Eds.), The handbook of discourse analysis, (pp. 538-547). Oxford: Blackwell.
Shohamy, E. (1995). Performance assessment in language testing. Annual Review of Applied Linguistics, 15, 188-211.
Sydorenko, T., Maynard, C., & Guntly, E. (2015). Rater behaviour when judging language learners’ pragmatic appropriateness in extended discourse. TESL Canada Journal, 32(1), 19-41.
Taguchi, N. (2011). Rater variation in the assessment of speech acts. Pragmatics, 21(3), 453-471.
Tajeddin, Z., & Alemi, M. (2014). Criteria and bias in native English teachers’ assessment of L2 pragmatic appropriacy: Content and FACETS analyses. The Asia-Pacific Education Researcher, 23(3), 425-434.
Tajeddin, Z., Alemi, M., & Razzaghi, S. (2014). Cross-cultural perceptions of impoliteness by native English speakers and EFL learners: The case of apology speech act. Journal of Intercultural Communication Research, 43(4), 304-326.
Tajeddin, Z., & Alemi, M. (2015). Functional communication in English. Tehran: Jungle Press.
Weigle, S. C. (1994b). Effects of training on raters of ESL compositions. Language Testing, 11(2), 197-223.
Weigle, S. C. (2002). Assessing writing. Cambridge: Cambridge University Press.
Xi, X. (2007). Evaluating analytic scoring for the TOEFL Academic Speaking Test (TAST) for operational use. Language Testing, 24(2), 251-286.
Youn, S. J. (2007). Rater bias in assessing the pragmatics of KFL learners using facets analysis. Second Language Studies, 26(1), 85-163.
Iranian Journal of Applied Language Studies
Volume 11, Issue 2
October 2019
Pages 31-56

Files

Share

How to cite

Statistics