Educational Effectiveness Research

About five decades of educational effectiveness research (EER) have brought the topic of edu-cational effectiveness to a prominent position in research agendas around the world. While in-itial research results in this area indicated that “Schools bring little influence to bear on a child’s achievement that is independent of his background and general social context” (Coleman et al., 1966, p. 325), currently there is a widespread consensus among researchers that schools influ-ence children’s development and educational outcomes in many ways (Chapman et al., 2015;

Reynolds et al., 2014; Teddlie & Reynolds, 2000).

3.1.1 Strands of educational effectiveness research

Depending on the underlying research interest, three major strands of EER can be distinguished:

School Effects Research, that studies the scientific properties of school effects; Effective Schools Research, that focuses on the processes of effective schooling and is initially often based on qualitative case studies of well-performing outlier schools; and School Improvement Research, that examines how schools can be changed and improved over time (Teddlie & Reyn-olds, 2000).

School Effects Research is concerned with the influence of schooling on intended student out-comes. Good and Brophy (1986) define school effects as what is known about the ability of schools to affect the outcomes of the students that they serve. A similar definition is given by Raudenbush and Willms (1995, p. 308), who define school effects as “…the extent to which attending a particular school modifies a student’s outcome.” The underlying question here is to what extent the school environment shows a separate influence on student outcomes beyond certain input characteristics of the student body. School effects essentially focus on the identi-fication of factors which enhance effectiveness in the school environment using methodological sound approaches.

The development of the School Effects Research branch also can be seen as a reaction to the Coleman Report, which concluded that “…the inequalities imposed in children by their home, neighborhood, and peer environment are carried along to become the inequalities with which they confront adult life at the end of the school” (Coleman et al., 1966, p. 325). In addition to

the pessimistic conclusion drawn concerning the influence of school-related factors, researchers also tried to address or counter methodological concerns that were brought up regarding EER (at that time called school effectiveness research) from the very beginning. Teddlie and Reyn-olds, for example, state that the Coleman Report received many criticisms about methodologi-cal issues, including the charge “that they did not operationalize the school input variables ad-equately in order to properly assess the effect that schools have on student achievement” (Ted-dlie & Reynolds, 2000, p. 58). Researchers in this field are predominantly concentrating on general methodological and psychometric issues such as reliability, generalizability, or validity.

The second branch of EER, the Effective Schools Research, also emerged as a reaction to the Coleman Report. Research in this strand initially tried to refute results from the report, and intended to prove that schools can do and make a difference. Focus here is set on the identifi-cation of highly successful schools and students, and comparing them with comparable schools – in terms of student composition – that are less effective in terms of student outcomes. In that sense, it can be argued that “A more effective school is one in which student performance is higher than predicted by input” (Chapman et al., 2015, p. 27). The research interest in Effective Schools Research is mainly focused on identifying differences between schools in order to un-derstand the conditions that lead to more effective schools. Research designs are usually based on qualitative case studies of especially effective schools and originally focused mainly on pub-lic schools attended by children from low socio-economic backgrounds – for example, Ed-monds (1979).

School Improvement Research, however, is not primarily focused on detecting effectiveness-enhancing factors related to outcome variables, but rather seeks to develop strategies to enable schools to become more effective. The main focus here is on change processes in educational contexts that should be described and ideally improved. Here, the individual school is consid-ered the center of the change – thus, changes and reforms need to consider the internal condi-tions of a school, and usually to follow a systematic approach of improvement over several years. Hopkins (2001, p. 13) defines school improvement as a “distinct approach to educational change that aims to enhance student outcomes as well as strengthening the school’s capacity for managing change”.

The current research project seeks to detect effectiveness-enhancing factors in the GCC coun-tries and aims to describe the relationship among them; it therefore is based in the school effects research paradigm.

3.1.2 International comparative studies and educational effectiveness

The historical context described in the previous section predominantly reflects the develop-ments of research in a Western context, where most of the research was done. Teddlie and Reynolds (2000, p. 232) argued that educational effectiveness in the past “has shown heavily ethnocentric tendencies” and they found, when evaluating the corresponding literature, that re-search in this field is “almost exclusively based upon scholars and rere-searchers within the coun-try of origin of the writer.” They therefore concluded that “the area of international effective-ness research…suggests an area so far relatively undeveloped.” Most of the literature in the past stems from Western countries, mainly from North America, Great Britain, The Netherland, Canada, Australia, Norway and Sweden. More than a decade later, Reynolds et al. (2014, p. 221) still emphasize the importance of the international dimension, stressing in their state-of-the-art review of EER that “An international perspective is of vital importance, since EER (Educational effectiveness research) may not mean the same thing in different parts of the world.”

Thus far, only a few studies, such as the International School Effectiveness Research Project (ISERP) as described by Reynolds (2006), have explicitly adopted a research design to measure educational effectiveness. The study was conducted in nine educational systems, but among them only two from outside the Western Hemisphere – namely Hong Kong and Taiwan. In spite of major differences across countries and especially between Western and Asian school systems, Reynolds also reported important similarities in terms of the factors that are associated with good schools: “We cannot stress too highly that many factors that make for good schools are conceptually quite similar in countries that have widely different cultural, social, and eco-nomic contexts. The factors hold true at the school level, but the detail of how school-level concepts play out within countries is different between countries. At the classroom level, the powerful elements of expectation, management, clarity, and instructional quality transcend cul-ture” (Reynolds, 2006, pp. 554–555).

Postlethwaite and Ross (1992) were among the first to use the vast range of contextual variables contained in international large-scale assessments to identify indicators associated with a kind of educational effectiveness. They analyzed data from the IEA Reading Literacy study con-ducted between 1989 and 1992, which included 32 educational systems from all over the world – but none of them in the Gulf area. Summarizing their results, schools associated with higher achievement tended to be well-managed, initiative-taking, well-stocked with library books, and had teachers who were more professional and used particular methods of teaching (encouraging

the students to read, emphasizing assessments, having high demands on structure, and so forth).

Unfortunately, this study ignored the hierarchical structure of the data by not disentangling the effects of different educational levels (for example by applying hierarchal multi-level analyses) – a critical consideration for this type of analysis as argued by Raudenbush and Bryk (1986).

Martin, Mullis, Gregory, Hoyle, and Shen (2000) based their analyses on the TIMSS 1995 study and included data from 34 educational systems, but again, at that time, no Gulf State partici-pated in the study. Their contribution can be seen as one of the first studies to use international large-scale assessments while concurrently taking the hierarchical structure of the data into ac-count. Martin et al. found that factors related to the socio-economic status (SES) of the student distinguished more uniformly between high- and low-achieving schools across countries than factors that are more directly related to the school, class, and teacher level. Subsequently, a growing number of authors applied multilevel modeling techniques to account for the clustering effects of nested data when using IEA TIMSS and PIRLS data for analyses in the field of edu-cational effectiveness (for example Kyriakides, 2006; Lamb & Fullarton, 2001; Rutkowski &

Rutkowski, 2008; Schwippert, 2001; Webster & Fisher, 2000).

The above-mentioned authors, among others, focused on what occurs within schools and tried to identify “value-added” variables by investigating characteristics related to organization, form and content. Findings from previous multilevel analyses of the author using eighth Grade data of TIMSS 2007 and 2003 (Neuschmidt, Hencke, Rutkowski, & Rutkowski, 2010; Neuschmidt, Hencke, Rutkowski, & Rutkowski, 2011) indicated home background indicators and nationality status as the most important predictors of mathematics achievement in the Gulf area. In addi-tion, different class- and school- level related variables, such as student behavior, teaching ex-perience, and monitoring homework were found to be significant indicators predicting mathe-matics outcomes.

Results from PISA 2012 indicated the following major general findings on system level (OECD, 2013): A negative relation between stratification in school systems and equity; and a more equitable allocation of school resources as well as a greater degree of school autonomy in terms of curricula and assessment in high-performing countries. Results also indicated lower performance for systems with larger proportions of students who arrive late for school and skip classes. In PISA 2012, the two GCC countries (Qatar and the United Arab Emirates) that par-ticipated, together with 63 other educational systems, showed a country mean in mathematics achievement far below the OECD average – but also, interestingly, greater equity concerning

their educational outcomes. Looking at the PISA 2015 results (OECD, 2016a), a similar pattern also can be discerned for both countries in science achievement.

An investigation on educational effectiveness factors was also among the research topics pre-sented in the Relationships Report published by the International Study Center for TIMSS &

PIRLS (Martin & Mullis, 2013). For the 34 countries and three benchmarking participants that administered TIMSS and PIRLS to the same students, the relationship between school, teacher, and home background scales on one hand, and student achievement in the three subjects on the other, were analyzed. For this purpose, several two-level hierarchical linear models were con-structed. The sampling design of most of the analyzed educational systems didn’t allow for the creation of three-level models; given the usual selection of one class per school, the variance components between schools and classes could not be separated. While the authors found con-siderable differences across countries concerning the achievement levels between schools and in the relation of school variables to student achievement, the results between the three subjects were found to be very similar. The home resources indicator was found to be the most important predictor for achievement. After controlling for the home background on both levels, the school environment scales indicating school safety/orderliness and emphasis on academic success still played an important role in many of the analyzed countries. The most important school instruc-tional scale was found to be the student engagement in reading, mathematics, and science. In all four GCC countries (Qatar, Oman, Saudi Arabia, and United Arab Emirates) for which data was available, student engagement was significantly associated with achievement even after controlling for the home background. Other important predictors emerging in the region were:

schools are safe and orderly and school support for academic success.

It is important to note here that the concepts used to measure effectiveness might differ in dif-ferent regions of the world, and may therefore not necessarily reflect the Western view which often mainly focuses on academic achievement. Harber and Muthukrishna (2000, p. 430), in-vestigating school effectiveness in South African schools in the 1990s, for example, describe an ideological dimension of effectiveness aimed at “fostering a non-violent, non-racist and democratic society” which goes beyond dimensions of functional effectiveness that include in-dicators like an orderly atmosphere and businesslike behavior.

In a seminar related to TIMSS Mathematics Learning Outcomes in Doha (Qatar) the author asked representatives from Ministries of Education and National Committees for Education from Qatar, Bahrain, and Kuwait about their definition of an effective school in their country.

Participants listed the following characteristics, summarized in the report from Khan (2015):

An effective school:

• Is capable of achieving its future vision for education in light of the international vi-sion.

• Guarantees distinguished and equal educational opportunities for all and thus helps students achieve better than expected results.

• Helps the students to acquire positive trends related to citizenship.

• Cares about teachers’ career development.

• Offers opportunities for participation, teamwork and fruitful cooperation amongst teachers.

• Provides modern educational resources for the students and the teachers.

• Provides diversified technological systems.

• Provides assessments and agendas.

• Caters for all students’ inclinations and trends in school activities.

This list of characteristics of effective schools resulting from the TIMSS seminar also shows a certain emphasis on educational quality and equity, which reflect the main dimensions of edu-cational effectiveness regarded in the West. However, beyond the focus on academic outcomes, respondents introduced the idea that effective schooling in the region is also required to deliver a good civic education in the sense that students should, as a result, become good citizens of their country. Such statements point to the importance of an additional function of schooling in the region: namely, legitimization of the respective system of government. The different func-tions of schooling are described in the section on educational quality in chapter 3.2.