Detecting Emotion as a Construct Irrelevant Factor in Reading Section of TOEFL

Document Type : مقالات علمی پژوهشی

Authors
Y. Kheyrkhahnia 1
B. Ghonsooly 2
J. Salehi Fadardi 3
1 Ph.D. Candidate of English teaching, Department of English, Qeshm Branch, Islamic Azad University, Qeshm, Iran
2 Department of English, Qeshm Branch, Islamic Azad University, Qeshm, Iran/ Professor of Applied Linguistics, Department of English, Ferdowsi University of Mashhad, Iran
3 Department of English, Qeshm Branch, Islamic Azad University, Qeshm, Iran / Professor of Psychology, Department of Psychology, Ferdowsi University of Mashhad, Iran
Abstract
Understanding the impact of various factors on language testing is important. Therefore, it is necessary to understand how they affect test scores in order to design and standardize language tests (Bachman, 1990). Based on the same logic and considering the need of fair reviewing for any tests claimed by ETS (2010), it is essential to identify, reduce and eliminate factors unrelated to the structure that hinder the optimal performance of test takers (Messick, 1989).

According to Vinson (2014), words themselves are a powerful tool for expressing emotions. Does a factor such as vocabularies in a text stimulate emotional reactions? When we read a text, we use our knowledge to understand its vocabularies, but in addition, by reading the vocabularies, emotion may be stimulated, which are deliberately not examined during the test. This research is intended to investigate if the vocabularies of TOEFL passages are likely to stimulate emotion as the construct-irrelevant factor which could affect the accuracy and legitimacy of the TOEFL test. Also, our hypothesis is that the amount of emotion evoked in the words of TOEFL texts is different from each other, and other factors intervene in this category.

By using the initial pilot sampling and with the help of PASS Software to determine the size of the final sample, 393 people were randomly selected by Random Number Generator Software. In addition, according to the method of detecting emotions by forming an emotional dictionry proposed by Turney (2002), the present study labled parts of speech of each word and then the words were grouped together as meaningful expressions into a five-page list of phrases like a dictionary. Through this method, the present study could determine the intensity and valenance of the selected particpants’ emotions in relation to the phrases selected from TOEFL iBT reading passages. Also, it was significant to measure if emotional intelligence could be influential on the evoked emotions of the words and phrases; thus, Emotional Intelligence Questionare of Schutte (1998) was selected. The research procedure was that the selected participants read three TOEFL passages without answering their reading questions; instead, they did Emotional Intelligence Questionnaire and self-reported their emotions through the five-page list of words and phrases.

The results of this study confirmed the validity of the research hypotheses, in the sense that not only the words and phrases of the three TOEFL passages in this research caused emotions but also the three passages were different in terms of emotion, and it can be argued that this can be a construct-irrelevant factor when reading and comprehending TOEFL passages.

The results of this study can increase the awareness of TOEFL test designers. In other words, TOEFL test designers must consider the effect of emotional elements in language assessment because these elements may disrupt the mental order of test takers and can affect their performances. This study casts doubt on the validity and reliability of the TOEFL as a standard test. ETS (2010) is interested in a fair review of tests to identify and reduce factors unrelated to the structure so it is useful considering the emotional interactions in the process of assessing the validity and reliability of any tests.

Keywords

Subjects

Abdolrezapour, P., & Tavakoli, M. (2012). The relationship between emotional intelligence and EFL learners’ achievement in reading comprehension. Innovation in Language Learning and Teaching, 6(1), 1-13.
Allison P. Anderson, A. P., Mayer, M. D., Fellows, A. M., Cowan, D. R., Hegel, M. T., Buckey, J. C. (2017). Relaxation with Immersive Natural Scenes Presented Using Virtual Reality. Aerospace Medicine and Human Performance, 88(6), 520-526.
Alm, C. O., Roth, D., & Sproat, R. (Eds.). (2005, December). Emotions from text: Machine learning for text-based emotion prediction. Paper presented at Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Vancouver, Canada.
Arnold, M. B. (1960). Emotion and Personality. Psychological Aspects. New York: Columbia University Press.
Bachman, L. (1990) Fundamental Considerations in Language Testing. Oxford: Oxford University Press.
Berto, R. (2014). The role of nature in coping with psycho-physiological stress: a literature review on restorativeness. Behavioral Sciences, 4(4), 394-409.
Buckey, J. C. (2006). Space physiology. New York: Oxford University Press.
Ciarrochi, J., & Mayer, J. (2007). Applying emotional intelligence: A practitioners guide. New York: Psychology Press.
Daiyan, M. D., Tiwari, S. K., Kumar, M., Alam, M. A. (2015). A Literature Review on Opinion Mining and Sentiment Analysis. International Journal of Emerging Technology and Advanced Engineering, 5(4), 619-623.
Das, S. & Chen, M. (2001). Yahoo! for Amazon: Extracting market sentiment from stock message boards. Proceedings of the Asia Pacific Finance Association annual Conference. Bangkok: Thailand.
ETS. (2009). The Official Guide to the TOEFL Test. New York: McGraw-Hill
ETS. (2010). TOEFL iBTTM research insight: TOEFL iBTTM test framework and test development. TOEFL iBTTM research, 1(1), 1-10.
Gernsbacher, M. A. & Robertson, R. R. (1992). Knowledge activation versus sentence mapping when representing fictional characters’ emotional states. Language and Cognitive Processes, 7(3-4), 353-371.
Gross, J. J. (2007). Handbook of emotion regulation. New York: The Guilford Press.
Hovy, E. H. (2015). What are sentiment, affect, and emotion? Applying the methodology of Michael Zock to sentiment analysis. Language Production, Cognition, and the Lexicon, 13– 24.
Kissler J., Assadollahi R., & Herbert C. (2006). Emotional and semantic networks in visual word processing: Insights from ERP studies. Progress in Brain Research, 156,147-183.
Kuchinke L., Jacobs, A.M., Gubrich, C., Võ, M.L.H., Conrad, M., & Herrmann, M. (2005). Incidental effects of emotional valence in single word processing: An fMRI study. NeuroImage, 28, 1022-1032.
Lanzetta, J. T., Sullivan, D. G., Masters, R. D., & McHugo, G. J. (1985). Viewers' emotional and cognitive responses to televised images of political leaders. In S. Kraus & R. Perloff (Eds.), Mass media and political thought (pp. 85-116). Beverly Hills, CA: Sage.
Laszlo, J., & Cupchik, G. C. (1995). The role of affective processes in reading time and time experience during literary experience. Empirical Studies of the Arts, 13, 25-37.
Liu, B. (2012). Sentiment analysis and opinion mining. San Rafael, California: Morgan & Claypool Publishers.
Liu, B. (2015).Sentiment Analysis: mining opinions, sentiments, and emotions. Cambridge: Cambridge University Press.
Mar, R. A., Oatley, K., Djikic, M., & Mullin, J. (2011). Emotion and narrative fiction: Interactive influences before, during, and after reading. Cognition & Emotion, 25(5), 818-833.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13-103). New York: Macmillan.
Mikolajczak, M., & Luminet, O. (2008). Trait emotional intelligence and the cognitive appraisal of stressful events: An exploratory study. Personality and Individual Differences, 44, 1445-1453.
Miller, G. A., Beckwith, R., Fellbaum, C., Gross, D., Miller, K. J. (1990). Introduction to WordNet: An on-line lexical database. International Journal of Lexicography, 3(4), 235-244.
NematTabrizi,A. R., & Esmaeili, L. (2016). The relationship between the emotional intelligence and reading comprehension of Iranian EFL impulsive vs. reflective students. International Journal of English Linguistics, 6(6), 221-229.
Peng, W. & Park, D. H. (2011). Generate Adjective Sentiment Dictionary for Social Media Sentiment Analysis Using Constrained Nonnegative Matrix Factorization. In L. A. Adamic, R. A. Baeza-Yates & S. Counts (Eds.), The international conference on weblogs and social media: Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media (pp.273- 280). Spain: Barcelona.
Pishghadam, R. & Poor-Esfahani, A. F. (2017). Introducing Emotioncy as an Effective Tool for the Acceptance of Persian Neologisms. Language Related Research, 8(5), 79-105.
Pishghadam, R. (2009). A Quantitative Analysis of the Relationship between Emotional Intelligence and Foreign Language Learning. Electronic Journal of Foreign Language Teaching, 6(1), 31-41.
Pishghadam, R. (2015). “Emotioncy in language education: From exvolvement to involvement”. Paper presented at the 2nd conference of Interdisciplinary Approaches to Language Teaching, Literature, and Translation Studies. Iran, Mashhad.
Pishghadam, R., Baghaei, P., & Seyedenozadi, Z. (2016). Introducing emotioncy as a potential source of test bias: A mixed rasch modeling study. International journal of Testing, 16, 1-14.
Poria, S., Cambria, E., Bajpai, R., Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98-125.
Saito, Y., Garza, T.J., & Horwitz, E.K. (1999). Foreign language reading anxiety. Modern Language Journal, 83, 202-218.
Salovey, P. and Mayer, J. D. (1990). Emotional Intelligence. Imagination, Cognition and Personality, 9(3), 185-211.
Scherer, K. R., Schorr, A., & Johnstone, T. (2001). Series in affective science. Appraisal processes in emotion: Theory, methods, research. New York, NY: Oxford University Press.
Schutte, N. S., Malouff, J. M., Hall, L. E., Haggerty, D. J., Cooper, J. T.., Golden, C. J., & Dornheim, L. (1998). Deveopment and validation of a measure of emotional intelligence. Personality and Individual Differences, 25, 167-177.
Shahghasemi, E., Heisey, D. R., & Mirani, G. (2011). How do Iranians and U.S. Citizens perceive each other: A systematic review. Journal of Intercultural Communication, 27, 1-10
Siakaluk, P. D., Ian Newcombe, P. Duffels, B., Li, E., Sidhu, D. M., Yap, M. J., & Pexman, P. M. (2016). Effects of emotional experience in lexical decision. Frontiers in Psychology, 7, 1-14.
Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the conference on empirical methods in natural language processing (1631-1642). Seattle: Washington.
Strauss, G. P., & Allen, D. N. (2008). Emotional intensity and categorization ratings for emotional and nonemotional words. Cognition and Emotion, 22(1), 114-133.
Turney, P.D. (2002) Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews. In P. Isabelle (Ed.), Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (pp.417-424). Pennsylvania: Philadelphia.
Van Dijk, T. A. (2002). Political discourse and political cognition. Politics as text and talk, 7, 203_237.
Vega, M. (1996). The representation of changing emotions in reading comprehension. Cognition & Emotion, 10(3), 303-322.
Vessel E.A., & Russo S. (2015). Effects of reduced sensory stimulation and assessment of countermeasures for sensory stimulation augmentation. Hanover (MD): NASA.
Vinson, D., Ponari, M., & Vigliocco, G. (2014). How does emotional content affect lexical processing? Cognition Emotion, 28, 737-746.
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales. Journal of Personality and Social Psychology, 54, 1063-1070.
Way, B. M., & Masters, R. D. (1996). Political attitudes: Interactions and affect. Motivation and Emotion, 20(3), 205- 236.
Language Related Research
Volume 11, Issue 4 - Serial Number 58
September and October 2020
Pages 105-128