The Study of the Content Validity Supportive Factors of the EPT Multiple-Choice Reading Comprehension Test: Based on Componential Perspective

Document Type : مقالات علمی پژوهشی

Authors
R. Rezaei 1
F. Aziz Malayeri 2
A. Bayat 2
H. Ahmadi 2
1 Ph.D. candidate, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran
2 Assistant Professor of TEFL, Department of English Language, Malayer Branch, Islamic Azad University, Malayer, Iran
10.29252/LRR.13.4.22
Abstract
The validity supporting factors of the EPT reading comprehension test, a standard language proficiency test held monthly in Iran, were investigated in this research paper. To this end, each of the test questions was assigned to one of the known effective linguistic and non-linguistic components used by different readers in comprehending a reading text. Our ability to read in a second language, at first, depends on our familiarity with linguistic knowledge, which is the words and grammar of the target language, and under the influence of our non-linguistic knowledge; that if the assessment of reading comprehension ability is unfairly mixed with our "non-linguistic knowledge", the result will not reflect the content validity of the test. Since high-stake tests like EPT are designed for examinees with different experiences and other intrapersonal cognitive abilities, balancing the content of the test will lead us to more favorable results. 85 questions from 22 expository texts were randomly selected, and 150 female students graduated from the theoretical fields of humanities, experimental, and mathematics from Malayer city participated in the test. The data were analyzed using Pearson's correlation statistical analysis, Heterotrait-monotrait ratio, Exploratory Factor Analysis, and the Partial Least Square model. The results showed that non-linguistic knowledge has a significant and more than expected effect on the performance of the testees. Teachers, lecturers, and institutes implementing and designing second language reading comprehension standardized tests will use the results of this research.



1. Introduction

To determine the level of proficiency in a foreign language, several language tests are frequently held each year. Language proficiency tests do not consider the method used by language learners; their emphasis is generally on the past language experience of the examinees and what they have learned until the time of the test (Jafarpour, 2015). Among the valid language tests held in Iran is the EPT, which has been held almost every month by the Islamic Azad University since 2012. EPT is composed of three parts: vocabulary, grammar, and comprehension. The importance of this exam is in the graduation of the students of this university at the PhD level.

In terms of test structure, EPT can be compared with TOEFL PBT. The only difference lies in the lack of listening and writing sections. This test consists of 100 questions; it does not have a negative score, and the time allotted is set on 140 minutes. The reading comprehension section has 35 questions, usually consisting of 4 separate expository texts on different fields. 15 questions are in the format of cloze test, and the rest is passage-based multiple choice questions.

The importance of evaluation, as stated by Black and William (1998), is undeniable both for teachers and learners. Therefore, a test content validity can guarantee the appropriateness of the result interpretations (Linn & Gronlund, 2000). Test validity can be interpreted as an agreement between the test score and the quality it is intended to measure (Kaplan & Saccuzzo, 2001). Thus, it can be concluded that any investigation and analysis on the effectiveness of the supporting content validity factors of a proficiency test can predict the adequacy of the level of linguistic knowledge as opposed to non-linguistic knowledge of the testees.

According to Bachman's language proficiency model (1990), strategic competence of language learners is based on their language competence and their knowledge structures, or their knowledge of the world (Alderson, Clapham, & Wall, 2005), and the knowledge they have acquired while learning their first language (Hedge, 2003). Here, it can be implied that theoretically, in order to be successful in reading comprehension tasks, in addition to language competence, one needs to be equipped with other reader-oriented variables such as synthesizing, infering, and evaluating information (Alderson, 2000).

As Nunan (1991) has stated, reading comprehension in a foreign language is different from that in one's first language. A word of caution hence should be mentioned in this regard. The assessment of this language ability should be run under a more controlled condition in terms of the testee's knowledge of the world. Nunan (1991) believed in the interaction of language competence and world knowledge when doing a comprehension task. Reading comprehension in a foreign language, as the result, needs separate sub-skills that are required to be taught (Alderson, 2000).

Considering the above mentioned premises, foreign language reading comprehension tests for mid-band language proficiency levels are expected to make use of these different source-based variables equally. This means that despite the inherent difficulty to assess this language ability, language-based questions should be paid the same attention as nonlanguage-based questions all over the test as possiblly as they can (Weir, 2005).

This parity is the motivation of the present study. Collecting intermediate passages employed in different EPT administrations, the study aimed at determining the load of different linguistic (text-dependent) and non-linguistic (reader-dependent) components of the test based on the componential perspective theories. The list of components is based on Farhadi and Hesami's (2005, p. 35) list of variables in their article, construct validity of L2 reading comprehension skills. The following research question is going to be answered:

Which group of components, linguistic versus non-linguistic components, has the strongest correlations with the value score obtained in the measurement of reading competence based on the gathered data?

2. Methodology

2.1. Subjects

A total number of 150 12th-grade female students took part in the present study. They were from three different fields of study (58 from experimental sciences, 48 from mathematical sciences, and 44 from humanities) from 3 language institutes in Malayer. They were selected by cluster random method. The subjects were between 18 to 22 years old. (M=19.5, SD=1.51). These students were intermediate language learners and got a score range between 51 and 70 out of 120 in the Oxford English Placement Test, so they were all placed at the CEFR level of B1.2 to B2.1.



2.2. Educational tools and materials

The material used in the present study consisted of 22 randomly-selected reading comprehension passages adapted from the EPT test. The test items underwent a close specification process to assign each item to the component it was intended to measure. The linguistic components understood vocabulary, using context, identifying facts and details, summarizing concepts, sequencing events, distinguishing between facts and opinions, and discovering main idea. For non-linguistic components, understanding point of view, determining author's purpose, drawing conclusions, identifying inferences, understanding cause and effect, using prior knowledge, identifying figurative language, and visualizing ideas were utilized.



3. Results

The exploratory factor analysis (EFA) was run in order to get a maximum dispersion of factor loadings within factors of linguistic and nonlinguistic components. For a smaller number of variables highly onto each factor resulting in more interpretable clusters of factors, principal axis factoring and varimax rotation methods were applied. SPSS results showed that the components of reading ability measured four constructs with an accuracy of 49.50 percent. From the results, scanning, and literal and reorganizational processing played an important role in comprehending second language intermediate texts of EPT.

The model of the standardized regression weights between each component and the overall linguistic and nonlinguistic constructs displayed in figure 1 indicates that the linguistic components of reading comprehension test had moderate and significant contributions to their construct, while nonlinguistic components of reading comprehension test had a high and significant contribusion to their construct.











Figure. 1, The relationships between components of constructs (standardized regression weights)

Keywords

Subjects

Abdelaal, N. M., & Sase, A. S., (2014). Relationship between prior knowledge and reading comprehension. Advances in Language and Literary Studies, 5(6), 125-131.
Adams, M. J., & Collins, A. (1977). A schema-theoretic view of reading comprehension (Report No. 032).Center for the Study of Reading Technical Report.
Amirian, M. R., Heshmatifar, Z., Zareian, Gh. & Davoudi, M. ((2018. Qualitative analysis of the factors affecting participation in English language preparation courses of the Ministry of Science, Research and Technology (MSRT): A data-driven approach. Linguistic Research in Foreign Languages, 8 (1), 27-56.
August, D., & Shanahan, T. (2010).Developing literacy in second-language learners: Report of the national literacy panel on language-minority children and youth. Journal of Literacy Research, 42(3), 341-348.
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
------------, Clapham C., & Wall, D. (2005). Language test construction and evaluation. Cambridge: Cambridge University Press.
Bachman, L. F. (1990). Fundamental considerations in language testing. New York: Oxford University Press.
Black, P., & William, D. (1998). Assessment and Classroom Learning: Assessment in Education. Assessment in Education, 5, 7-74.
Brame, C., & Biel, R. (2015). Test-enhanced learning: The potential for testing to promote greater learning in undergraduate science courses. CBE Life Sciences Education,14 (2), 1-12.
Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals. New York: Longman University Press.
Carr, J. F., & Harris, D. E. (2001). Succeeding with standards linking curriculum, assessment, and action planning. Alexandria Virginia: Association for Supervision and Curriculum Development.
Caldwell J. S. (2008). Comprehension assessment: A classroom guide. New York: Guilford Publication Press.
Chehrzad, M.H., & Ajideh, P. (2013). Effects of different response types on Iranian EFL test-takers' performance. Iranian Journal of Applied Language Studies, 5 (2), 29-50.
Coady, J. (1979). A psycholinguistic model for the ESL reader. In R. MacKay, B. Barkman, & R. R. Jordan (Eds.), Reading in a second language: Hypothesis, organization and practice (pp. 5-12).Newbury House.
Council of Europe. (2011). Manual for language test development and examining: ALTE. Strasburg. Language Policy Division.
Davis, A. (1968). Language testing symposium: A psycholinguistic approach. London: Oxford University Press.
Estaji, M. (2020). Does field of study matter in academic performance: Differential item functioning analysis of a high-stakes test using one-parameter and two-parameter item response theory models. Iranian Journal of English for Academic Purposes, 9(3), 14-31.
Farhadi, H., & Hessami, G. R. (2005). Construct validity of L2 reading comprehension skills. International Journal of Applied Linguistics.8(2), 29-53.
Fletcher, J. M. (2006). Measuring reading comprehension. Scientific Studies of Reading, 10, 323-330.
Garson, G. D. (2016). Partial least squares: Regression and structural equation models. Asheboro, NC: Statistical Associates Publishers.
Grabe, W. (1991). Current developments in second language reading research. TESOL Quarterly, 25, 375-406.
------------ (2009). Reading in a second language: Moving from theory to practice. Cambridge: Cambridge University Press.
Hair, J. F., Hult, G. T. M., Ringle, C., & Sarstedt, M. (2017). A primer on partial least squares structural equation modeling (PLS-SEM). Sage publications.
Hedge, T., (2003). Teaching & learning in the language classroom. UK: OUP.
Hemmati, S. J., Baghaei, P., Bemani, M. (2016). Cognitive diagnostic modeling of L2 reading comprehension ability: providing feedback on the reading performance of Iranian candidates for the university entrance examination. International Journal of Language Testing, 6(2), 92-100.
Hoover, W. A., & Tunmer, W. E. (1993). The components of reading. In G. G. Thompson, W. E. Tunmer, & T. Nicholson (Eds.), Reading acquisition processes (pp. 1-19). Multilingual Matters.
Jabbari, S., Khademi, M. (2013). Creating a diagnostic test for reading comprehension of elementary school students. Curriculum Research, 3 (2), 33-51.
Jafarpour, A. J. (1996). Introduction to Language Testing. Shiraz: Shiraz University Press.
Kaplan, R. M., & Saccuzzo, D. P. (2001). Psychological testing: Principles, applications, and issues (5th Ed.). Belmont, CA: Wadsworth.
Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York: Cambridge University Press.
Laufer, B. (1992). Reading in a foreign language: How does L2 lexical knowledge interact with the reader’s general academic ability?. Journal of Research in Reading, 15(1), 95-103.
Linn, R. L., & Gronlund, N. E. (2000). Measurement and assessment in teaching (8th Ed.). Columbus, OH: Merrill, Prentice Hall.
Lunzer, E., Waite, M., & Dolan, T. (1979). Comprehension and comprehension test. In E. Lunzer & K. Gardner (Eds.), The effective use of reading (pp. 37-71). Heinemann Educational Books Ltd.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (pp. 13-103). Macmillan.
Munby, J. (1978). Communicative syllabus design. Cambridge: Cambridge University Press.
Nunan, D. (1991). Language teaching methodology. London: Prentice-Hall.
Palomba, C. & Banta, T.W. (1999) Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass Publishers.
Paris, S. G., & Hamilton, E. E. (2009). The development of children’s reading comprehension. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp.32-53). Routledge.
Pasquarella, A., Gottardo, A., & Grant, A.(2012). Comparing factors related to reading comprehension in adolescents who speak English as first (L1) or second (L2) languages. Scientific Studies of Reading, 1(6), 1-29.
Perfetti, C. (1985). Reading ability. New York: Oxford University Press.
--------------, Landi, N., & Oakhill, J. (2005). The acquisition of reading comprehension skills. In M. J. Snowling & C. Hulme (Eds.), The science of reading: a handbook (pp. 227-247). Blackwell.
-------------- (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 8, 293-304.
Quellette, G. P., (2006). What’s meaning got to do with it: The role of vocabulary in word reading and reading comprehension. Journal of Educational Psychology. 98(3), 554-566.
RAND Reading Study Group. (2002). Reading for understanding: Towards an R&D program in reading comprehension. OERI.
Ranjbaran, F. & Alavi, S. M. (2017). Developing a reading comprehension test for cognitive diagnostic assessment: A RUM analysis .Studies In Educational Evaluation, 55, 167-179.
Rogde, K., Hagen, A. M., Melby-Lervag, M., & Lervag, A. (2019). The effect of linguistic comprehension instruction on generalized language and reading comprehension skills: A systematic review. Campbell Systematic Reviews, 15(4), 1-37.
Roohani, A., Dayeri, K., & Farhang Ju, M. (2017). Role of background knowledge in Iranian EFL learners' reading comprehension test performance. International Journal of Language Testing, 7(1), 28-39.
Rumelhart, D. (2004). Toward an interactive model of reading. In R. B. Rudell & N. J. Unrau (Eds.), Theoretical models and processes of reading. (pp. 1149-1179). International Reading Association.
Sahraei, R., & Mamaghani, H. (2012). Evaluation of the validity and reliability of the English language test of the Ministry of Science, Research and Technology. Educational Measurement Quarterly, 3 (10), 1-20.
Schoonen, R., Hulstijn, J., & Bossers, B. (1998). Meracognitive and language-specific knowledge in native and foreign language reading comprehension: An empirical study among Dutch students in grades 6, 8 and 10. Language Learning, 48(1), 71-106.
Shirvan, M., Ghonsooly, B., & Fatemi, A. (2016). The Effectiveness of strategy-based instruction in teaching English as a second or foreign Language: A meta-analysis of experimental studies. US-China Foreign Language, 14(3), 163-181.
Shahmirzadi, N., Siyyari, M., Marashi, H., & Geramipour, M. (2020). Test fairness analysis in reading comprehension PhD nationwide admission test items under CDA. Journal of Foreign Language Research,10(1), 152-165.
Sheybani, E., & Zeraatpishe, M. (2018). On the dimensionality of reading comprehension tests composed of text comprehension items and cloze test items. International Journal of Language Testing, 8(1), 12-26.
Sinclair, J. (1991). Corpus, concordance, collocation. Oxford: Oxford University Press.
Skehan, P. (1998). A cognitive approach to language learning. Oxford: Oxford University Press.
Stovich, K. E. (2000). Progress in understanding reading: Scientific foundations and new frontiers. New York: Guilford Press.
Urquhart, A. H., & Weir, C. J. (1998). Reading in a second language: Process, product, and practice. New York: Longman.
Vandergrift, L., & Goh, C. C. M. (2012). Teaching and learning second language listening: Metacognition in action. New York: Routledge.
Van der Walt, J. L., & Steyn, H. S. (2008). The validation of language tests. Stellenbosch Papers in Linguistics, 38, 191-204.
Van Dijk, T. A. (1977). Text and context: Explorations in the semantics of text. London: Longman.
Weir, C. J. (2005). Language test validation: An evidence-based approach. Oxford: Palgrave.
Zandi, H., Keivanpanah, S., & Alavi, S. M.(2014).The effect of test specifications review on improving the quality of a test. Iranian Journal of Language Teaching Research, 2(1), 1-14.
Zarei, A. A. & Shakoori Nia, S. (2014). The effect of vocabulary, syntax, and discourse-oriented activities on short and long-term L2 reading comprehension. International Journal of Language and Linguistics, 1(1), 30-39.
Zhang, D. (2012). Vocabulary and grammar knowledge in second language reading comprehension: A structural equation modelling study. The Modern Language Journal, 96(4), 558-575.
Language Related Research
Volume 13, Issue 4 - Serial Number 70
September and October 2022
Pages 727-762