Parvaneh Shayeste-Far; Gholam Reza Kiany; Monireh Norouzi
Abstract
1. IntroductionTesting-based reform has often been used as a lever for introducing desirable changes. This tenet is found in light of the claim that what is targeted to be measured by testing, in particular by tests of higher stakes, would strongly determine and shape what gets taught and learnt at the ...
Read More
1. IntroductionTesting-based reform has often been used as a lever for introducing desirable changes. This tenet is found in light of the claim that what is targeted to be measured by testing, in particular by tests of higher stakes, would strongly determine and shape what gets taught and learnt at the lower stakes. Clear examples are various University Entrance Examinations (UEEs) which have long appeared to have immense importance for the people and institutions involved. In fact, high-stakes tests such as the UEEs are widely perceived to have the potential to shape curricular teaching and learning. Such a test consequential effect, i.e., ‘test washback’, has encouraged decision makers and educators to manage for a test-based change to promote the quality of education in general and the quality of learning and teaching in particular. Reform policies as such have also been increasingly recognized by policymakers, educators, and research community in Iran, during the past decade. Such a recognition and appreciation of reform policies consequently led to reform initiatives that aimed at enhancing the quality of high school learning and teaching through reforming the conventional UEEs system of the country. The outcome was the ‘Act of Student Admission at Universities (ASAU)’ which mandates gradual replacement of the English requirements of the UEEs with high school National English Achievement Tests (NEATs) students have to take over their four high school years. However, the existing evidence shows that success is not necessarily ensured as hoped. Reasons for such a failure were mostly studied after ‘implementation’ or at the ‘completion phase’ of a reformed test but not during its ‘implementation phase’. Such an early evaluation has been mostly neglected. Taking into account the perceptions and perspectives of the English Language Teaching (ELT) practitioners of the country, the present study aimed to explore the effectiveness of the UEEs program change in improving the ELT context of the Ministry of Education (ME). 2. MethodologyA mixed-methods approach was employed to collect both qualitative and quantitative data. For such a purpose, semi-structured interviews with 16 ELT professors and head-teachers followed by questionnaire data from English language teachers were used. Participants were instructors of the Ministry of Science, Research and Technology (MSRT) and ME. Eight university professors and 8 ME teachers, selected through purposive sampling, participated in the qualitative phase, and 406 English teachers (both males and females, with teaching experience of 5 to 30 years), selected through cluster and convenient sampling procedures, participated in different phases of the study. The interview guide was prepared on the basis of the information obtained from the Act, higher-order policy documents of the country, available literature, and the researchers’ experience and knowledge of the two admission systems. The interview guide questions sought interviewees’ views about the consequences of the new tests on the ME’s English Foreign Language (EFL) teaching and learning. The interviews were tape recorded, transcribed, and then carefully content-analyzed.Next, quantitative data were collected through the researcher-made ‘Questionnaire of Change Consequences-Teachers’ version (QCC-Ts)’ that was distributed among 406 English teachers. The interview analyses together with a review of the related literature gave more than one hundred categories and behaviors showing high-stakes test impacts (e.g., teachers’ considerations of practicing the skills measured by high-stakes tests) which later led to the development of a temporary driven model of change impact stated through the QCC-Ts, with a total of 82 items forming three main scales: Teachers’ Planning and Instruction, Learning Objectives, and Fairness in admission. A series of factor analyses, both Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA) were run to substantiate the whole scale. 3. Discussion3.1. Interviews: Professors and Head-TeachersData analysis of qualitative phase resulted into the identification of more than 200 themes that were finally subsumed under 14 categories. The necessity of “Adequacy of the new admission criteria”, “fairness in assessment through the new program”, “Students’ quality and depth of knowledge” as well as considerations about the “quality of the output of the ME”, and “Resurgence of the similar coaching for the UEEs” were the most commonly mentioned areas perceived to be negatively influenced by the NEATs-based program. A relative dissatisfaction was found in the interviewees’ views on the final outcome of the NEATs program in terms of its effects on the quality, quantity, rate, and depth of learning. In teaching area, the participants reported a probability for integrating a balanced teaching and learning of the main skills of the English language in the ELT contexts of high schools. However, the respondents found the consequences positive on “students’ learning activities” and “teachers’ assessment of students’ learning”. That is, the new program will lead to more student engagement in learning activities, and increased regular assessment by teachers. 3.2. Questionnaire Surveys: English TeachersThe results of the factor analysis of the QCC-Ts revealed a significantly good-fitting model for the constructed scales (CMIN/DF=2.79, RMSEA=0.04, CFI=0.981, GFI=0.984, and a small χ2=42). “Classroom activities and time-arrangement”, and “teaching materials” teachers would use in their classes were perceived as the two areas being most strongly driven by the new high-stakes tests (X=4.40, SD=.96; X=4.35, SD=.79 for classroom activities and materials, respectively). For instance, perceived consequences of the NEATs on ‘types and structures of classroom activities’ ‘students’ test-taking skills and strategies’, ‘extra time spent on grammar instruction’, and on ‘particular classroom activities purposefully adopted (e.g., vocab., translation, and structure drills) to increase students’ scores in the NEATs’ were reported by mean scores clustering around 4.5 in the Likert scale that illustrates ‘agreement’ or near ‘complete agreement’ with the NEATs impacts. Inconsistency between the desired policies and the observed practices was found for teachers’ ‘methods of teaching’ and ‘content of teaching’ that came up with the mean values of 4.01 and 3.59, respectively. For instance, two-thirds of the participants said that they change their teaching methods in order to ‘develop students’ NEAT-taking skills’ or ‘increase students’ scores in these tests’. These show how measurement-driven program of the ME obliges EFL teachers to change their adopted methods, employed activities, and applied materials in the direction of meeting testing objectives rather than learning objectives. As to the NEATs consequences on learning-related factors, EFL teachers were almost neutral about the NEATs consequences on learning-related factors, especially with regard to ‘affective consequences’ (2.5
Reza Kiyani
Abstract
IntroductionIn the current era of policies and practices of testing-based accountability, high- stakes tests such as university entrance exams are widely perceived to have immense importance for the people and institutions involved because they induce a rather equal curriculum through a renewed focus ...
Read More
IntroductionIn the current era of policies and practices of testing-based accountability, high- stakes tests such as university entrance exams are widely perceived to have immense importance for the people and institutions involved because they induce a rather equal curriculum through a renewed focus on what is measured. In fact, high-stakes tests have the potential to shape curricular teaching and learning. Such a consequential alignment of teaching and learning with testing, i.e., ‘test wash back’ has recently encouraged policymakers to manage for a reform in situations where high-stakes tests can be deliberately employed to promote standards of teaching, accountability and powerful learning. Macro policies as such have formed the key concerns of many reformers around the globe, including Iran. A scrutiny of the ways these policies are conceptualized at their planning phase, and the possible dilemmas and challenges anticipated for their implementation have been the subject of very scarce studies, however. The present study aimed to unveil macro-policies, plans, values, and conceptualizations underlying different perspectives of a community of policymakers and planners planning for gradual substitutions of the University Entrance Examinations (UEEs) with the High school National Achievement Examinations (HNAEs) and students’ academic background, in a test change context in Iran.2. MethodologyThe present study, based on in-depth interviews with 14 high-rank policymakers and proponents of the UEEs reform, detailed their conceptualization of this change in terms of the underlying policies, prospects, and perspectives. The participants enjoyed different levels of experience in education (management, evaluation, and teaching) and age range of 40 to 55. An interview guide was developed for the qualitative nature of the required data. Strauss and Corbin’s (1990) ‘Paradigm Model’ of qualitative data analysis was used as a tool for identifying thematic categories and subcategories. This particular model is a data-driven conceptual model that works based on a series of causal/consequential relationships among the categories or themes.3. DiscussionThe initial database yielded a template which revealed participants’ understanding of the given situation, the logic underpinning their planning, their examinations of the problems, and the prospects about the programme’s future. Three major themes finally emerged: (1) the induction of the intended consequences; (2) the value of the multiple-approach assessment of learners’ knowledge and abilities; and (3) the significance of the prerequisites and challenges for implementation: current and future trends. (1)According to all respondents, what formed the main rationale behind the UEEs reform was to counterbalance the negative impacts of the objective UEEs and trigger intended positive effects on curriculum, instruction, and learning via the HNAEs. They felt that such potentials can be actualized through opting for ‘systemic validity’ which is defined by Fredriksen and Collins (1989) as a process that sparks off constructive positive influences on teaching and learning. Most of the inter viewees ,in general, believed improving both quality and quantity of teaching and learning can be fostered through changing the UEEs-based programme to the one which values schooling instructions and aligns the assessment modes/means with constructivism concepts in education. The conceptual picture of ‘Consequences’ does not limit only to the systemic validity but it embraces test-related factors like fairness or psychometric characteristics too. The governmental policies based on which the HNAEs were and are built come from the fairness and social equity premises. In relation to measuring real abilities through fair measures, the respondents questioned psychometric traditions for decision-making about the candidates’ abilities. They all converged in their beliefs that the UEEs with their sizable proportion of the memorization-based items are not perceived as fair measures for selecting the students. (2) All emerging themes and subthemes revealed a progress moving away from a ‘measurement culture’ that limits students’ performance to tightly specific skills captured under specific times towards ‘edumetrics culture’ (Segers & Dochy, 2001).The themes ‘integrating qualitative measurement modes like the interviews or oral communications for specific majors (e.g., English Language majors or arts)’, ‘keeping an on-going record of students’ performances from the beginning of their high school towards their graduation’, ‘exploiting regular formative assessments rather than conventional summative ones per se’, ‘integrating IT in assessing students’ learning’, and ‘designing and administering standard tests of Educational Progress (like SAT for instance) several times rather than once a year’ support this assertion. (3) Informants’ descriptions raised their shared concerns about the provision of the logistics ranging from allocation of financial, material, and human resources to timely collaboration and communication between the two ministries of Education and Science, Research and Technology (MSRT )and the National Organization of Educational Testing(NOET). Besides these requirements, the data came up with other concerns that would make serious challenges in future. They also explained their doubts in ‘discrimination power’ of the HNAEs (0-20) compared to the standardized UEEs, or argued for the likelihood of distributing a compatible ‘test-anxiety and stress’ over the four years of high schools.4. Conclusion
In Iran, choosing between the two competing admission practices by the policy makers bears a testimony that policymakers’ tendency to shift to the directing function of the HNAEs resulted from dynamics of their power. It is, then, within the realm of such a power that the unintended washback associated with the selecting function of the high-stakes tests seems to be controlled. Motivated by the current debates on evaluating the changed programmes, this study contributes to the literature through exploring the planning/policy phase prior to a later evaluation of the sole products which is common to the traditional evaluations. Policy/planning phase analyses can establish a baseline for subsequent evaluation of any programme, revealing a constellation of factors that might mitigate the intended policies, visions, or missions of that programme. In this study, a partial congruence was found between the policy and the desired outcomes of the HNAEs programme. This may thwart the success rate of ideals intended by the underpinning policies. Such concerns are not unique to Iran, but in other systems as well.
