Volume 13, Issue 4 (2022)
LRR 2022, 13(4): 727-762 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Rezaei R, Aziz Malayeri F, Bayat A, Ahmadi H. The Study of the Content Validity Supportive Factors of the EPT Multiple-Choice Reading Comprehension Test:
Based on Componential Perspective. LRR 2022; 13 (4) :727-762
URL: http://lrr.modares.ac.ir/article-14-49847-en.html
URL: http://lrr.modares.ac.ir/article-14-49847-en.html
1- Ph.D. candidate, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran
2- Assistant Professor of TEFL, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran , faramarz.amalayeri@hotmail.com
3- Assistant Professor of TEFL, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran
2- Assistant Professor of TEFL, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran , faramarz.amalayeri@hotmail.com
3- Assistant Professor of TEFL, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran
Abstract: (1074 Views)
The validity supporting factors of the EPT reading comprehension test, a standard language proficiency test held monthly in Iran, were investigated in this research paper. To this end, each of the test questions was assigned to one of the known effective linguistic and non-linguistic components used by different readers in comprehending a reading text. Our ability to read in a second language, at first, depends on our familiarity with linguistic knowledge, which is the words and grammar of the target language, and under the influence of our non-linguistic knowledge; that if the assessment of reading comprehension ability is unfairly mixed with our "non-linguistic knowledge", the result will not reflect the content validity of the test. Since high-stake tests like EPT are designed for examinees with different experiences and other intrapersonal cognitive abilities, balancing the content of the test will lead us to more favorable results. 85 questions from 22 expository texts were randomly selected, and 150 female students graduated from the theoretical fields of humanities, experimental, and mathematics from Malayer city participated in the test. The data were analyzed using Pearson's correlation statistical analysis, Heterotrait-monotrait ratio, Exploratory Factor Analysis, and the Partial Least Square model. The results showed that non-linguistic knowledge has a significant and more than expected effect on the performance of the testees. Teachers, lecturers, and institutes implementing and designing second language reading comprehension standardized tests will use the results of this research.
1. Introduction
To determine the level of proficiency in a foreign language, several language tests are frequently held each year. Language proficiency tests do not consider the method used by language learners; their emphasis is generally on the past language experience of the examinees and what they have learned until the time of the test (Jafarpour, 2015). Among the valid language tests held in Iran is the EPT, which has been held almost every month by the Islamic Azad University since 2012. EPT is composed of three parts: vocabulary, grammar, and comprehension. The importance of this exam is in the graduation of the students of this university at the PhD level.
In terms of test structure, EPT can be compared with TOEFL PBT. The only difference lies in the lack of listening and writing sections. This test consists of 100 questions; it does not have a negative score, and the time allotted is set on 140 minutes. The reading comprehension section has 35 questions, usually consisting of 4 separate expository texts on different fields. 15 questions are in the format of cloze test, and the rest is passage-based multiple choice questions.
The importance of evaluation, as stated by Black and William (1998), is undeniable both for teachers and learners. Therefore, a test content validity can guarantee the appropriateness of the result interpretations (Linn & Gronlund, 2000). Test validity can be interpreted as an agreement between the test score and the quality it is intended to measure (Kaplan & Saccuzzo, 2001). Thus, it can be concluded that any investigation and analysis on the effectiveness of the supporting content validity factors of a proficiency test can predict the adequacy of the level of linguistic knowledge as opposed to non-linguistic knowledge of the testees.
According to Bachman's language proficiency model (1990), strategic competence of language learners is based on their language competence and their knowledge structures, or their knowledge of the world (Alderson, Clapham, & Wall, 2005), and the knowledge they have acquired while learning their first language (Hedge, 2003). Here, it can be implied that theoretically, in order to be successful in reading comprehension tasks, in addition to language competence, one needs to be equipped with other reader-oriented variables such as synthesizing, infering, and evaluating information (Alderson, 2000).
As Nunan (1991) has stated, reading comprehension in a foreign language is different from that in one's first language. A word of caution hence should be mentioned in this regard. The assessment of this language ability should be run under a more controlled condition in terms of the testee's knowledge of the world. Nunan (1991) believed in the interaction of language competence and world knowledge when doing a comprehension task. Reading comprehension in a foreign language, as the result, needs separate sub-skills that are required to be taught (Alderson, 2000).
Considering the above mentioned premises, foreign language reading comprehension tests for mid-band language proficiency levels are expected to make use of these different source-based variables equally. This means that despite the inherent difficulty to assess this language ability, language-based questions should be paid the same attention as nonlanguage-based questions all over the test as possiblly as they can (Weir, 2005).
This parity is the motivation of the present study. Collecting intermediate passages employed in different EPT administrations, the study aimed at determining the load of different linguistic (text-dependent) and non-linguistic (reader-dependent) components of the test based on the componential perspective theories. The list of components is based on Farhadi and Hesami's (2005, p. 35) list of variables in their article, construct validity of L2 reading comprehension skills. The following research question is going to be answered:
Which group of components, linguistic versus non-linguistic components, has the strongest correlations with the value score obtained in the measurement of reading competence based on the gathered data?
2. Methodology
2.1. Subjects
A total number of 150 12th-grade female students took part in the present study. They were from three different fields of study (58 from experimental sciences, 48 from mathematical sciences, and 44 from humanities) from 3 language institutes in Malayer. They were selected by cluster random method. The subjects were between 18 to 22 years old. (M=19.5, SD=1.51). These students were intermediate language learners and got a score range between 51 and 70 out of 120 in the Oxford English Placement Test, so they were all placed at the CEFR level of B1.2 to B2.1.
2.2. Educational tools and materials
The material used in the present study consisted of 22 randomly-selected reading comprehension passages adapted from the EPT test. The test items underwent a close specification process to assign each item to the component it was intended to measure. The linguistic components understood vocabulary, using context, identifying facts and details, summarizing concepts, sequencing events, distinguishing between facts and opinions, and discovering main idea. For non-linguistic components, understanding point of view, determining author's purpose, drawing conclusions, identifying inferences, understanding cause and effect, using prior knowledge, identifying figurative language, and visualizing ideas were utilized.
3. Results
The exploratory factor analysis (EFA) was run in order to get a maximum dispersion of factor loadings within factors of linguistic and nonlinguistic components. For a smaller number of variables highly onto each factor resulting in more interpretable clusters of factors, principal axis factoring and varimax rotation methods were applied. SPSS results showed that the components of reading ability measured four constructs with an accuracy of 49.50 percent. From the results, scanning, and literal and reorganizational processing played an important role in comprehending second language intermediate texts of EPT.
The model of the standardized regression weights between each component and the overall linguistic and nonlinguistic constructs displayed in figure 1 indicates that the linguistic components of reading comprehension test had moderate and significant contributions to their construct, while nonlinguistic components of reading comprehension test had a high and significant contribusion to their construct.
Figure. 1, The relationships between components of constructs (standardized regression weights)
1. Introduction
To determine the level of proficiency in a foreign language, several language tests are frequently held each year. Language proficiency tests do not consider the method used by language learners; their emphasis is generally on the past language experience of the examinees and what they have learned until the time of the test (Jafarpour, 2015). Among the valid language tests held in Iran is the EPT, which has been held almost every month by the Islamic Azad University since 2012. EPT is composed of three parts: vocabulary, grammar, and comprehension. The importance of this exam is in the graduation of the students of this university at the PhD level.
In terms of test structure, EPT can be compared with TOEFL PBT. The only difference lies in the lack of listening and writing sections. This test consists of 100 questions; it does not have a negative score, and the time allotted is set on 140 minutes. The reading comprehension section has 35 questions, usually consisting of 4 separate expository texts on different fields. 15 questions are in the format of cloze test, and the rest is passage-based multiple choice questions.
The importance of evaluation, as stated by Black and William (1998), is undeniable both for teachers and learners. Therefore, a test content validity can guarantee the appropriateness of the result interpretations (Linn & Gronlund, 2000). Test validity can be interpreted as an agreement between the test score and the quality it is intended to measure (Kaplan & Saccuzzo, 2001). Thus, it can be concluded that any investigation and analysis on the effectiveness of the supporting content validity factors of a proficiency test can predict the adequacy of the level of linguistic knowledge as opposed to non-linguistic knowledge of the testees.
According to Bachman's language proficiency model (1990), strategic competence of language learners is based on their language competence and their knowledge structures, or their knowledge of the world (Alderson, Clapham, & Wall, 2005), and the knowledge they have acquired while learning their first language (Hedge, 2003). Here, it can be implied that theoretically, in order to be successful in reading comprehension tasks, in addition to language competence, one needs to be equipped with other reader-oriented variables such as synthesizing, infering, and evaluating information (Alderson, 2000).
As Nunan (1991) has stated, reading comprehension in a foreign language is different from that in one's first language. A word of caution hence should be mentioned in this regard. The assessment of this language ability should be run under a more controlled condition in terms of the testee's knowledge of the world. Nunan (1991) believed in the interaction of language competence and world knowledge when doing a comprehension task. Reading comprehension in a foreign language, as the result, needs separate sub-skills that are required to be taught (Alderson, 2000).
Considering the above mentioned premises, foreign language reading comprehension tests for mid-band language proficiency levels are expected to make use of these different source-based variables equally. This means that despite the inherent difficulty to assess this language ability, language-based questions should be paid the same attention as nonlanguage-based questions all over the test as possiblly as they can (Weir, 2005).
This parity is the motivation of the present study. Collecting intermediate passages employed in different EPT administrations, the study aimed at determining the load of different linguistic (text-dependent) and non-linguistic (reader-dependent) components of the test based on the componential perspective theories. The list of components is based on Farhadi and Hesami's (2005, p. 35) list of variables in their article, construct validity of L2 reading comprehension skills. The following research question is going to be answered:
Which group of components, linguistic versus non-linguistic components, has the strongest correlations with the value score obtained in the measurement of reading competence based on the gathered data?
2. Methodology
2.1. Subjects
A total number of 150 12th-grade female students took part in the present study. They were from three different fields of study (58 from experimental sciences, 48 from mathematical sciences, and 44 from humanities) from 3 language institutes in Malayer. They were selected by cluster random method. The subjects were between 18 to 22 years old. (M=19.5, SD=1.51). These students were intermediate language learners and got a score range between 51 and 70 out of 120 in the Oxford English Placement Test, so they were all placed at the CEFR level of B1.2 to B2.1.
2.2. Educational tools and materials
The material used in the present study consisted of 22 randomly-selected reading comprehension passages adapted from the EPT test. The test items underwent a close specification process to assign each item to the component it was intended to measure. The linguistic components understood vocabulary, using context, identifying facts and details, summarizing concepts, sequencing events, distinguishing between facts and opinions, and discovering main idea. For non-linguistic components, understanding point of view, determining author's purpose, drawing conclusions, identifying inferences, understanding cause and effect, using prior knowledge, identifying figurative language, and visualizing ideas were utilized.
3. Results
The exploratory factor analysis (EFA) was run in order to get a maximum dispersion of factor loadings within factors of linguistic and nonlinguistic components. For a smaller number of variables highly onto each factor resulting in more interpretable clusters of factors, principal axis factoring and varimax rotation methods were applied. SPSS results showed that the components of reading ability measured four constructs with an accuracy of 49.50 percent. From the results, scanning, and literal and reorganizational processing played an important role in comprehending second language intermediate texts of EPT.
The model of the standardized regression weights between each component and the overall linguistic and nonlinguistic constructs displayed in figure 1 indicates that the linguistic components of reading comprehension test had moderate and significant contributions to their construct, while nonlinguistic components of reading comprehension test had a high and significant contribusion to their construct.
Figure. 1, The relationships between components of constructs (standardized regression weights)
Keywords: EPT reading comprehension test, CEFR intermediate band, linguistic components, non-linguistic components
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |