The Optimal Cognitive Diagnostic Model (CDM) for the Grammar Section of MA Entrance Examination of State Universities for EFL Candidates

Document Type : مقالات علمی پژوهشی

Authors
M. Geramipour 1
H. Talebzadeh 2
S. Mahdi 3
1 Assistant Professor of Curriculum Studies, Faculty of Psychology & Education, Kharazmi University, Tehran, Iran
2 Assistant Professor of Applied Linguistics, Department of Foreign Languages, Faculty of Literature & Humanities, Kharazmi University, Tehran, Iran.
3 Ph.D. Candidate in Higher Education Management, Faculty of Management, Kharazmi University, Tehran, Iran
10.29252/LRR.12.1.6
Abstract
One of the most innovative tools for researchers in order to improve the quality of education and assessment is the application of Cognitive Diagnostic Approaches (CDA) which is the result of the unification of cognitive psychology and educational measurement; unfortunately, they are scarcely utilized by (grammar) language education and assessment researchers (in Iran). Among the most important decisions to be made in the process of cognitive diagnostic assessment are determining the (sub-)skills required to respond correctly to each question as well as choosing an optimal cognitive diagnostic model. The present qualitative and quantitative study aims to develop a Q-matrix in order to identify such cognitive (sub-)skills, to determine the optimal cognitive-diagnostic model (CDM) for the grammar section of MA entrance examination for English majors, and to pinpoint mastery and non mastery states of the examinees who were 5000 MA entrance participants. The findings from the literature review, informants’ and experts’ evaluations, and think-aloud protocols in the Q-matrix development phase revealed that each MA examination grammar item taps into between two to four of the six attributes of verb tense, correct verb usage, idiomatic expressions, modifiers, agreement, and preposition. Evaluation of three alternative models [i.e. the Deterministic-Input, Noisy-And (DINA), Deterministic Input, Noisy-Or-gate (DINO), and Generalized DINA (GDINA)] from among the existing cognitive diagnostic models indicated that G-DINA was the best fit for the Grammar data. Considering the performance of master and non-master participants, the study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing



1. Introduction

Cognitive diagnostic assessment (CDA) is designed to measure specific cognitive skills of students, so as to provide information about their cognitive strengths and weaknesses (Leighton & Gierl, 2007). Previous research on CDA in language testing mostly focused on reading comprehension and listening sections, whereas less attention has been paid to grammar. Moreover, in most of the previous studies (Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Lee & Sawaki, 2009; Ravand, 2015; Ravand, Barati & Widhiarso, 2013) just a specified cognitive diagnostic model (CDM) was fitted to the language test data while searching for an optimum CDM was generally overlooked.

Given the importance of high-stakes tests such as university entrance examinations, the current research aims to apply CDA to an Iranian high-stakes English grammar test to specify the underlying skills required to answer the test items correctly; furthermore, it intends to detect strengths and weaknesses of the students based on the identified skills. In doing so, searching for an optimum CDM was adopted to find the best fitting model to the second language grammar test data.



2. Literature Review

Cognitive diagnostic models are confirmatory multidimensional latent variable models with complex structures. These models let researchers propose exact hypotheses about the nature of cognitive processes that students use in response to test items (Rupp et al., 2010). Three classes of saturated, compensatory, and non-compensatory models of CDA are available to researchers. That is, a- the saturated model titled generalized deterministic inputs, noisy “and” gate (G-DINA), b-compensatory models (e.g., the Deterministic Input Noisy Output “OR” gate (DINO) and the additive CDM (ACDM)) which allow for other skills to contribute to the chance of responding to an item correctly, and c- the non-compensatory models (e.g., The Deterministic Input Noisy Output “AND” gate (DINA) and the reduced reparametrized unified model (RRUM)), where possessing all cognitive skills is necessary to answer a test item correctly (de la Torre, 2011).

In recent years, many CDM studies were conducted on data from different fields of studies including language assessment (Alavai & Ranjbaran, 2018; Alderson et al., 2015; Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Li, 2011; Lee & Sawaki, 2009; Minaei et al., 2014; Moghadam et al., 2015; Park & Cho, 2011; Ranjbaran & Alavi, 2016; Ravand, 2015; Ravand et al., 2013; Yie, 2016, 2017). Although Park and Cho (2011) applied CDA on the English grammar for Korean EFL learners, only Yie (2017) searched for an Optimal Cognitive diagnostic model in a second language grammar test data. The latter study is very similar to the present study in many respects but the eventual cognitive diagnostic model.



3. Methodology

A retrofitting approach (Jang, 2009) to CDA was adopted to reach the cognitive diagnostic model. In doing so, at the first stage of the CDM, a Q matrix (de la Torre, 2011) was qualitatively developed based on the findings from the literature review, the viewpoints of an English language expert panel, and the cognitive processes extracted from college students' think-aloud protocols. The resulting Q-matrix provided all of the required skills needed to answer all of the grammar test items of the Iranian MA entrance examination for English majors. Then, DINA, DINO, and the G-DINA models were empirically fitted to the grammar test data of 5000 participants through the CDM package of R (George et al., 2016).



4. Conclusion

The findings of this study showed that the saturated G-DINA model was the best fitting model for the grammar data. The compensatory DINO model also fitted the data, yet the non-compensatory DINA model did not fit the grammar item response data based on the absolute model fit indices.

In line with the results of Park and Cho (2011), this study also confirms that the six underlying skills including 1-verb tense, 2- correct verb usage, 3- idiomatic expressions, 4- modifiers, 5- agreement, and 6- preposition encompass almost all of the required grammar skills. Moreover, verb tense skill was identified as the weakness of the students, while idiomatic expressions skill was a strength point. Altogether, in a second language context, it seems that even the students majoring in the English language do not master all of the required skills of grammar. The study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing

Keywords

Subjects

Alavi, M. & Ranjbaran, F. (2018). Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analysis of a Reading Comprehension Test Battery. Journal of English Language Teaching and Learning, 21, pp. 1-31.
Alderson, J. C, Brunfaut, T., & Harding, L. (2015). Towards a Theory of Diagnosis in Second and Foreign Language Assessment: Insights from Professional Practice across Diverse Fields. Applied Linguistics, 36(2), 236-260.
Azar, B. S., & Hagen, S. A. (2017). Understanding and using English grammar. Hoboken, NJ : Pearson Education.
Baghaei, P., & Ravand, H. (2015). A cognitive processing model of reading comprehension in English as a foreign language using the linear logistic test model. Learning and Individual Differences, 43, 100-105.
Cavanaugh, J. E. (2009). 171: 290 Model Selection, Lecture VI: The Baysian Information Criterion. Department of Biostatistics, Department of Statistics and Acturial Science, University of Iowa.
Clark, A. K. (2013). Validation of a cognitive diagnostic model across multiple forms of a reading comprehension assessment (Doctoral dissertation).Retrieved from https://kuscholarworks.ku.edu/bitstream/handle/1808/15071/Clark_ku_0099D_13062_DATA_1.pdf?sequence=1.
DiBello, L. V., Roussos, L. A., & Stout, W. (2007). Review of cognitive diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics, Vol. 26: Psychometrics (pp. 979 - 1030). Elsevier Science B.V.: The Netherlands.
Feng, Y. (2013). Estimation and Q-matrix validation for diagnostic classification models, Master Thesis Submitted to the University of South California. Retrieved form https://scholarcommons.sc.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=3630&context=etd
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Ps
ychological Bulletin, 76(5), 378–382. doi:10.1037/h0031619
Griffin, P. & Nix, P. (1991). Educational Assessment and reporting, A new approach. New South Wales: Harcourt Brace Jovanovich.
Hartman, H. J. (2001). Developing students’ metacognitive knowledge and skills. In H.J. Hartman (Ed.). Metacognition in learning and instruction: Theory, Research and Practice (pp. 33-68). Dordrecht, the Netherlands: Kluwer.
Hartz, S. M.; Roussos, L. and Staut, W. (2002). Skills Diagnisus: Theory and Practice. Computer Software User Manual for Arpeggio Software. Princeton, NJ: ETS.
Jang, E. E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for Fusion Model application to LanguEdge assessment. Language Testing, 26(1), 31-73.
Koizum, R. Sakai, H., Ido, T., Ota, H., Hayama, M., Sato, M., & Nemoto, A. (2011). Development and validation of a diagnostic grammar test for Japanese learners of English. Language Assessment Quarterly, 8(1), 53-72.
Kunina-Habenicht, O.; Rupp, A. A. and Wilhelm, O. (2009). A practical illustration of multidimensional diagnostic skills profiling: Comparing results from confirmatory factor analysis and diagnostic classification models, Studies in Educational Evaluation, Volume 35, Issues 2–3 pp. 64-70, Retrieved on May 2018 from https://www.sciencedirect.com/science/article/pii/S0191491X09000224
Landis, J., & Koch, G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach. Journal of Educational Measurement, 41(3), 205-237.
Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239-263. DOI: 10.1080/15434300903079562
Li, H. (2011). Evaluating language group differences in the subskills of reading using a cognitive diagnostic modeling and differential skill functioning approach. Doctoral dissertation. Penn State University, State College, PA. Available online at https://etda.libraries.psu.edu/files/final_submissions/2272
Li, H., Hunter, C. V., & Lei, P.-W. (2015, online first). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing. Doi: 10.1177/0265532215590848
Liao, Y. F. (2007). Investigating the Construct Validity of the Grammar and Vocabulary Section and the Listening Section of the ECCE: Lexico-Grammatical Ability as a Predictor of L2 Listening Ability. Working Papers in Second or Foreign Language Assessment, Volume 5. Retrieved on June 2017 from https://michiganassessment.org/wp-content/uploads/2014/12/Spaan_V5_FULL.pdf#page=45
Lim, Y. (2015). Cognitive Diagnostic Model Comparisons. PhD Dissertation submitted to the Georgia Institute of Technology.
Madison M., Bradshaw L. (2013). The effects of Q-matrix design on classification accuracy in the LCDM. Rocky Hill, CT.
Mislevy, R. J. (2006). Cognitive psychology and educational assessment. Educational measurement, 4, 257-305.
Murphy, R. (2015). Essential grammar in use: A self-study reference and practice book for elementary students of English: with answers. Delhi: Cambridge University Press.
Park, Chanho and Cho, Sookyung (2011). Cognitive Diagnostic Assessment of English Grammar for Korean EFL Learners. English Teaching, 66(4), 101- 117.
Ravand, H., Barati, H., & Widhiarso, W. (2013). Exploring Diagnostic Capacity of a High Stakes Reading Comprehension Test: A Pedagogical Demonstration. Iranian Journal of Language Testing, 3(1).
Ravand, Hamdollah & Robitzsch, Alexander (2015). Cognitive Diagnostic Modeling Using R Practical Assessment, Research & Evaluation, 20(11). Available online: http://pareonline.net/getvn.asp?v=20&n=11
Rupp, A., Templin, J. & Henson, R. (2010). Diagnostic measurement: theory, methods, and applications. New York: The Guilford Press.
Tatsuoka, K. K. (1995). Architecture of knowledge structure and cognitive diagnosis: A statistical pattern recognition and classification approach. In P. D. Nichols, S. F. Chipman, and R. L. Brennan (Eds.), Cognitively Diagnostic Assessment, pp. 327-361 .
Thomson, A. J., & Martinet, A. V. (2012). A practical English grammar. Oxford: Oxford Univ. Press.
Wu, X. (2003). Intrinsic Motivation and Young Language Learners: the Impact of the Classroom Environment. System, 31, 501 – 517
Yi, Y.S. (2016). Probing the relative importance of different attributes in L2 reading and listening comprehension items: An application of cognitive diagnostic models. Langauge Testing, 34 (3), 337-355.
Yi, Y.S. (2017) In Search of Optimal Cognitive Diagnostic Model(s) for ESL Grammar Test Data, Applied Measurement in Education, 30(2), 82-101, DOI: 10.1080/08957347.2017.1283314.
Language Related Research
Volume 12, Issue 1 - Serial Number 61
March and April 2021
Pages 187-218