Volume 12, Issue 1 (2021)
LRR 2021, 12(1): 187-218 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Geramipour M, Talebzadeh H, Mahdi S. The Optimal Cognitive Diagnostic Model (CDM) for the Grammar Section of MA Entrance Examination of State Universities for EFL Candidates. LRR 2021; 12 (1) :187-218
URL: http://lrr.modares.ac.ir/article-14-34252-en.html
URL: http://lrr.modares.ac.ir/article-14-34252-en.html
1- Assistant Professor of Curriculum Studies, Faculty of Psychology & Education, Kharazmi University, Tehran, Iran , mgramipour@khu.ac.ir
2- Assistant Professor of Applied Linguistics, Department of Foreign Languages, Faculty of Literature & Humanities, Kharazmi University, Tehran, Iran.
3- Ph.D. Candidate in Higher Education Management, Faculty of Management, Kharazmi University, Tehran, Iran
2- Assistant Professor of Applied Linguistics, Department of Foreign Languages, Faculty of Literature & Humanities, Kharazmi University, Tehran, Iran.
3- Ph.D. Candidate in Higher Education Management, Faculty of Management, Kharazmi University, Tehran, Iran
Abstract: (3160 Views)
One of the most innovative tools for researchers in order to improve the quality of education and assessment is the application of Cognitive Diagnostic Approaches (CDA) which is the result of the unification of cognitive psychology and educational measurement; unfortunately, they are scarcely utilized by (grammar) language education and assessment researchers (in Iran). Among the most important decisions to be made in the process of cognitive diagnostic assessment are determining the (sub-)skills required to respond correctly to each question as well as choosing an optimal cognitive diagnostic model. The present qualitative and quantitative study aims to develop a Q-matrix in order to identify such cognitive (sub-)skills, to determine the optimal cognitive-diagnostic model (CDM) for the grammar section of MA entrance examination for English majors, and to pinpoint mastery and non mastery states of the examinees who were 5000 MA entrance participants. The findings from the literature review, informants’ and experts’ evaluations, and think-aloud protocols in the Q-matrix development phase revealed that each MA examination grammar item taps into between two to four of the six attributes of verb tense, correct verb usage, idiomatic expressions, modifiers, agreement, and preposition. Evaluation of three alternative models [i.e. the Deterministic-Input, Noisy-And (DINA), Deterministic Input, Noisy-Or-gate (DINO), and Generalized DINA (GDINA)] from among the existing cognitive diagnostic models indicated that G-DINA was the best fit for the Grammar data. Considering the performance of master and non-master participants, the study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing
1. Introduction
Cognitive diagnostic assessment (CDA) is designed to measure specific cognitive skills of students, so as to provide information about their cognitive strengths and weaknesses (Leighton & Gierl, 2007). Previous research on CDA in language testing mostly focused on reading comprehension and listening sections, whereas less attention has been paid to grammar. Moreover, in most of the previous studies (Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Lee & Sawaki, 2009; Ravand, 2015; Ravand, Barati & Widhiarso, 2013) just a specified cognitive diagnostic model (CDM) was fitted to the language test data while searching for an optimum CDM was generally overlooked.
Given the importance of high-stakes tests such as university entrance examinations, the current research aims to apply CDA to an Iranian high-stakes English grammar test to specify the underlying skills required to answer the test items correctly; furthermore, it intends to detect strengths and weaknesses of the students based on the identified skills. In doing so, searching for an optimum CDM was adopted to find the best fitting model to the second language grammar test data.
2. Literature Review
Cognitive diagnostic models are confirmatory multidimensional latent variable models with complex structures. These models let researchers propose exact hypotheses about the nature of cognitive processes that students use in response to test items (Rupp et al., 2010). Three classes of saturated, compensatory, and non-compensatory models of CDA are available to researchers. That is, a- the saturated model titled generalized deterministic inputs, noisy “and” gate (G-DINA), b-compensatory models (e.g., the Deterministic Input Noisy Output “OR” gate (DINO) and the additive CDM (ACDM)) which allow for other skills to contribute to the chance of responding to an item correctly, and c- the non-compensatory models (e.g., The Deterministic Input Noisy Output “AND” gate (DINA) and the reduced reparametrized unified model (RRUM)), where possessing all cognitive skills is necessary to answer a test item correctly (de la Torre, 2011).
In recent years, many CDM studies were conducted on data from different fields of studies including language assessment (Alavai & Ranjbaran, 2018; Alderson et al., 2015; Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Li, 2011; Lee & Sawaki, 2009; Minaei et al., 2014; Moghadam et al., 2015; Park & Cho, 2011; Ranjbaran & Alavi, 2016; Ravand, 2015; Ravand et al., 2013; Yie, 2016, 2017). Although Park and Cho (2011) applied CDA on the English grammar for Korean EFL learners, only Yie (2017) searched for an Optimal Cognitive diagnostic model in a second language grammar test data. The latter study is very similar to the present study in many respects but the eventual cognitive diagnostic model.
3. Methodology
A retrofitting approach (Jang, 2009) to CDA was adopted to reach the cognitive diagnostic model. In doing so, at the first stage of the CDM, a Q matrix (de la Torre, 2011) was qualitatively developed based on the findings from the literature review, the viewpoints of an English language expert panel, and the cognitive processes extracted from college students' think-aloud protocols. The resulting Q-matrix provided all of the required skills needed to answer all of the grammar test items of the Iranian MA entrance examination for English majors. Then, DINA, DINO, and the G-DINA models were empirically fitted to the grammar test data of 5000 participants through the CDM package of R (George et al., 2016).
4. Conclusion
The findings of this study showed that the saturated G-DINA model was the best fitting model for the grammar data. The compensatory DINO model also fitted the data, yet the non-compensatory DINA model did not fit the grammar item response data based on the absolute model fit indices.
In line with the results of Park and Cho (2011), this study also confirms that the six underlying skills including 1-verb tense, 2- correct verb usage, 3- idiomatic expressions, 4- modifiers, 5- agreement, and 6- preposition encompass almost all of the required grammar skills. Moreover, verb tense skill was identified as the weakness of the students, while idiomatic expressions skill was a strength point. Altogether, in a second language context, it seems that even the students majoring in the English language do not master all of the required skills of grammar. The study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing
1. Introduction
Cognitive diagnostic assessment (CDA) is designed to measure specific cognitive skills of students, so as to provide information about their cognitive strengths and weaknesses (Leighton & Gierl, 2007). Previous research on CDA in language testing mostly focused on reading comprehension and listening sections, whereas less attention has been paid to grammar. Moreover, in most of the previous studies (Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Lee & Sawaki, 2009; Ravand, 2015; Ravand, Barati & Widhiarso, 2013) just a specified cognitive diagnostic model (CDM) was fitted to the language test data while searching for an optimum CDM was generally overlooked.
Given the importance of high-stakes tests such as university entrance examinations, the current research aims to apply CDA to an Iranian high-stakes English grammar test to specify the underlying skills required to answer the test items correctly; furthermore, it intends to detect strengths and weaknesses of the students based on the identified skills. In doing so, searching for an optimum CDM was adopted to find the best fitting model to the second language grammar test data.
2. Literature Review
Cognitive diagnostic models are confirmatory multidimensional latent variable models with complex structures. These models let researchers propose exact hypotheses about the nature of cognitive processes that students use in response to test items (Rupp et al., 2010). Three classes of saturated, compensatory, and non-compensatory models of CDA are available to researchers. That is, a- the saturated model titled generalized deterministic inputs, noisy “and” gate (G-DINA), b-compensatory models (e.g., the Deterministic Input Noisy Output “OR” gate (DINO) and the additive CDM (ACDM)) which allow for other skills to contribute to the chance of responding to an item correctly, and c- the non-compensatory models (e.g., The Deterministic Input Noisy Output “AND” gate (DINA) and the reduced reparametrized unified model (RRUM)), where possessing all cognitive skills is necessary to answer a test item correctly (de la Torre, 2011).
In recent years, many CDM studies were conducted on data from different fields of studies including language assessment (Alavai & Ranjbaran, 2018; Alderson et al., 2015; Baghaie & Ravand, 2015; Clarck, 2013; Jang, 2009; Li, 2011; Lee & Sawaki, 2009; Minaei et al., 2014; Moghadam et al., 2015; Park & Cho, 2011; Ranjbaran & Alavi, 2016; Ravand, 2015; Ravand et al., 2013; Yie, 2016, 2017). Although Park and Cho (2011) applied CDA on the English grammar for Korean EFL learners, only Yie (2017) searched for an Optimal Cognitive diagnostic model in a second language grammar test data. The latter study is very similar to the present study in many respects but the eventual cognitive diagnostic model.
3. Methodology
A retrofitting approach (Jang, 2009) to CDA was adopted to reach the cognitive diagnostic model. In doing so, at the first stage of the CDM, a Q matrix (de la Torre, 2011) was qualitatively developed based on the findings from the literature review, the viewpoints of an English language expert panel, and the cognitive processes extracted from college students' think-aloud protocols. The resulting Q-matrix provided all of the required skills needed to answer all of the grammar test items of the Iranian MA entrance examination for English majors. Then, DINA, DINO, and the G-DINA models were empirically fitted to the grammar test data of 5000 participants through the CDM package of R (George et al., 2016).
4. Conclusion
The findings of this study showed that the saturated G-DINA model was the best fitting model for the grammar data. The compensatory DINO model also fitted the data, yet the non-compensatory DINA model did not fit the grammar item response data based on the absolute model fit indices.
In line with the results of Park and Cho (2011), this study also confirms that the six underlying skills including 1-verb tense, 2- correct verb usage, 3- idiomatic expressions, 4- modifiers, 5- agreement, and 6- preposition encompass almost all of the required grammar skills. Moreover, verb tense skill was identified as the weakness of the students, while idiomatic expressions skill was a strength point. Altogether, in a second language context, it seems that even the students majoring in the English language do not master all of the required skills of grammar. The study concludes with suggestions, implications, and applications of the findings for high-stakes language education and testing
Keywords: Cognitive diagnostic model (CDM), Optimal model, grammar, MA entrance examination, English language
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |