Educational Effectiveness Research (EER) – Definitions

An effective school:

• Is capable of achieving its future vision for education in light of the international vi-sion.

• Guarantees distinguished and equal educational opportunities for all and thus helps students achieve better than expected results.

• Helps the students to acquire positive trends related to citizenship.

• Cares about teachers’ career development.

• Offers opportunities for participation, teamwork and fruitful cooperation amongst teachers.

• Provides modern educational resources for the students and the teachers.

• Provides diversified technological systems.

• Provides assessments and agendas.

• Caters for all students’ inclinations and trends in school activities.

This list of characteristics of effective schools resulting from the TIMSS seminar also shows a certain emphasis on educational quality and equity, which reflect the main dimensions of edu-cational effectiveness regarded in the West. However, beyond the focus on academic outcomes, respondents introduced the idea that effective schooling in the region is also required to deliver a good civic education in the sense that students should, as a result, become good citizens of their country. Such statements point to the importance of an additional function of schooling in the region: namely, legitimization of the respective system of government. The different func-tions of schooling are described in the section on educational quality in chapter 3.2.

the literature and that these themselves are interrelated” (Creemers, Kyriakides, & Sammons, 2010, p. 4).

Educational Effectiveness

Many authors (for example Chapman et al., 2015; Scheerens, 2004b, 2016) see the basic func-tioning of the educational system as an Input – Output model that is influenced by process factors within, and by context factors external to, the system under consideration. An example of such a model is depicted in Figure 3-1. Accordingly, education can be seen as a production process (in the field of economics, this is also known as educational production function), man-aged by malleable inputs and processes, and ultimately leading to certain output factors, which are often measured on a student level (Scheerens, 2016). More details about the Input – Output model from the economic perspective and about the transfer from economic theory to the field of education can be found by Hanushek (1986). In organizational theory, a similar kind of model is referred to as the rational goal model, in which productivity and efficiency are the central criteria to assess effectiveness (Scheerens, 2004a, p. 124). Other models may emphasize different aspects of effectiveness: The open systems model focuses on growth and resource ac-quisition, while the human relations model focuses on human resource development, and the internal process model on stability and control (Scheerens, 2016).

Figure 3-1: Basic system model on the functioning of education (from Scheerens, 2016, p. 6) Following the input-process-output model approach, EER is primarily concerned with detecting malleable input and process variables that are associated with outcome factors of interest, often cognitive student outcomes (Creemers & Kyriakides, 2008; Scheerens, 2004b).

Context

Inputs Process or throughput outputs

System level School level classroom level

Scheerens therefore described effectiveness research as follows:

The major task of educational effectiveness research is to reveal the impact of relevant input characteristics on output and to

“break open” the black box in order to show which process or throughput factors “work”, next to the impact of contextual con-ditions (Scheerens, 2016, p. 6).

The term educational effectiveness tries to integrate a broad range of research areas from dif-ferent strands related to research on difdif-ferent levels, often with a focus on conditions on school level (such as school organization and policies) and classroom level (with a focus on teacher behavior, classroom instruction etc.; Chapman et al., 2015; Creemers & Kyriakides, 2008).

Scheerens (2016) also includes policy-amenable conditions at the national level.

Following advances in the field of effectiveness research in education, but also due to major improvements in the scientific methods applied, it is now generally accepted that those effec-tiveness-enhancing factors work on different levels of educational systems, while still being interrelated. When viewing educational systems as hierarchically organized, educational effec-tiveness can be regarded as an attempt to incorporate effeceffec-tiveness research from all different levels. The following broad definition of effectiveness research given by Creemers et al. (2010, p. 3) and quite similarly by Chapman et al. (2015) will be adopted for this thesis: “Education effectiveness research can be seen as an overarching theme that links together a conglomerate of research in different areas, including research on teacher behavior and its impacts; curricu-lum, student grouping procedures; school organization; and educational policy”.

According to the level of the educational system, EER can then be categorized into the follow-ing subareas:

• System effectiveness, a more recent term and not yet necessarily included in all defini-tions concerning educational effectiveness, was stimulated by the rise of international comparative assessments. It investigates malleable conditions at the national level that can be associated with student outcomes; for example, policies regarding to school au-tonomy, accountability, and choice of the school by parents.

• School Effectiveness then points to malleable factors on the school level, such as school organization and educational policies. Creemers and Kyriakides (2008, p. 3) give the following definition: “School effectiveness here refers to the role of school processes and organization: ‘the impact that school-wide factors, such as policy for

teaching, school climate, and the school’s perceived mission, have on student’s cogni-tive and affeccogni-tive performance.”

• On classroom level, the term teacher effectiveness focuses on the impact of teacher background and classroom factors on student performance. The terms instructional ef-fectiveness or teaching efef-fectiveness are sometimes used to specifically refer to activi-ties of the teachers in the classroom. These terms may partially be used interchangea-bly. In this thesis, the term teacher effectiveness will be used in the more general sense of Creemers and Kyriakides (2008, p. 3), who define teacher effectiveness as referring

“to the impact that classroom factors, such as teacher behavior, teacher expectations, classroom organization, and use of classroom resources, have on student perfor-mance.”

Value-added

While Chapman et al. describe EER similarly to the definitions above, they add an important value-added concept to the second part of their definition:

It therefore seeks to identify and explore the factors related to teaching, curriculum, and learning environments that may ex-plain in a statistical sense (both directly and indirectly) the vari-ation in student outcomes, while also controlling for student in-take characteristics such as socioeconomic status and prior attain-ment/prior ability (Chapman et al., 2015, p. 30).

The focus here is on effectiveness-enhancing factors that are purged from any context factors such as the student’s background. The term value-added is borrowed from the discipline of economics; when transferred to education, this concept basically means “a measure of the rel-ative gain in achievement made by pupils” (Teddlie & Reynolds, 2000, p. 264). The underlying idea is that a school or educational system is not so much responsible for the absolute level of the student achievement, but rather for the progress students make within the educational sys-tem. Consequently, any context factors that influence student achievement need to be disentan-gled as much as possible from the educational factors which often constitute the main focus of interest. According to this conceptualization “A more effective school is one in which student performance is higher than predicted by input” (Chapman et al., 2015, p. 27).

However, the extent to which a disentanglement of school instructional factors from back-ground factors is possible depends on different factors such as the data design (longitudinal or

cross-sectional), the availability of context factors regarded as important – such as prior ability or socio-economic status of the student, and the methodology used for the separation of educa-tional factors from context factors. Different value-added approaches and the model used for the current study will be discussed in more detail in section 8.2.2.

The OECD, after seeking expert input from the field, defined the value-added component of a school as “the contribution of a school to students' progress towards stated or prescribed edu-cation objectives (e.g. cognitive achievement). The contribution is net of other factors that con-tribute to students' educational progress” (OECD, 2008, p. 17).

Effectiveness and Quality Criteria

In the most general sense, effectiveness refers to the level of goal attainment; school effective-ness, therefore, refers to the school’s degree of achieving its educational objectives. This defi-nition, however, needs a clarification concerning the objectives of a school, who defines them, and how they can be measured. Usually, two general dimensions of effectiveness are discussed:

a quality dimension and an equity dimension.

Educational Quality:

According to Heid (2000), quality as such is not an objective and observable property of an object. Assigning a certain level of quality, therefore, only can be based on a subjective evalu-ation process. Heid argues that this evaluevalu-ation depends on explicit and implicit decisions about certain criteria to evaluate an objects’ nature, and that these decisions are made by those who claim to ensure and establish quality (Heid, 2000, p. 41). This means that it is not possible to define a uniform definition of educational quality, as due to the different interests of stakehold-ers involved in education, quality only can be defined on the base of a certain pstakehold-erspective (Har-vey & Green, 1993; Terhart, 2000). Har(Har-vey and Green (1993) define five major differing ceptualizations or categories of quality: quality as exceptional, quality as perfection or con-sistency, quality as fitness for purpose, quality as value for money, and quality as transfor-mation.

In EER, and hence in the scope of this thesis, the focus is mainly set on the aspect of the trans-formation of the participant. In that sense, “A quality education is one that affects changes in the participants and, thereby presumably enhances them” (Harvey & Green, 1993, p. 24), and these changes are usually measured against certain output criteria on a student or school level.

In consequence, quality here can be seen as the discrepancy between a desired outcome or

char-acteristic and a certain status or input condition based on a certain evaluation criterion. Educa-tional quality therefore depends on the objectives that an educaEduca-tional system or a school is supposed to fulfill, as they will define the output criteria. In this sense, the quality of structures (such as curriculum or opportunity to learn) and those of processes (which also could be eval-uated on their own) here are rather seen as effectiveness-enhancing factors determining the out-come quality (Creemers, 1994) .

Although historically there were several distinct modes of teaching and education (see also section 2.1 about the education in the Gulf Area), a modern kind of school system has become prevalent in current global trends, exhibiting similar characteristics and objectives. Adick (1992, p. 244) calls this the “universalization of modern schooling” and described universal common characteristics such as: a differentiated school system which distinguishes between classes, levels, and so forth; teaching according to a prearranged curriculum, professionalized staff teaching at scheduled time intervals; and state-controlled regulated educational practices in schools.

According to Adick, the objective of modern education systems can be seen as fulfilling certain qualification, selection, and legitimization functions:

The acquisition of sanctioned knowledge, rewarded with a certif-icate, becomes a form of cultural capital. This allocation of chances for a better life by means of the school seems to be basi-cally legitimate in the sense that everybody believes in it. And what is even more challenging for analysis, this model of school-ing is universally accepted (Adick, 1992, p. 244).

Fend (2006, p. 54) supported this notion of education systems being part of a universal project of modernity, and described four different social functions of an educational system and thus of a school: The qualification of the students; their allocation into the employment system, respectively into a social stratum; enculturation (referring to the reproduction of cultural capa-bilities and cultural comprehension of the world and the person); and the legitimization of the respective system of government.

The necessity of the qualification of the student body is obvious: for a society, it is important to have qualified members to make its economy competitive, and for an individual qualification

provides a better chance for good work conditions and high salaries. Fend (2006, p. 51) de-scribed job-related skills and knowledge therefore as the important educational outcomes on the level of the student body.

Nevertheless, questions remain regarding which criteria should be used to measure educational quality, and finally to determine the effectiveness of an institution or educational system ac-cording to the definitions listed above. While certain indicators, such as transitions to certain kinds of secondary education or university or the number of grade repetitions were initially used, it later was argued that decisions about promotion and referrals are influenced by other factors than education in a school or classroom alone (Creemers & Kyriakides, 2008). There-fore, cognitive criteria were preferred – mainly achievement in basic school subjects like math-ematics, reading, or science. Creemers and Kyriakides (2008, p. 20), when reviewing the effec-tiveness literature, consequently stated: “The majority of current studies collected data from national tests in subjects areas like mathematics and languages.” However, it can be argued that students in modern societies will need to learn more than basic skills in core subjects, leading in the direction of higher-order learning and metacognition. Levine and Lezotte (1990, p. 70), when reviewing achievement criteria to measure effectiveness, found that most tests assess

“fragmented, lower order skills.” While they acknowledged that these rather mechanical skills – such as basic computational skills – need to be mastered, especially in primary grades, they also argued for the necessity of including measures on higher order learning and thinking skills such as reading comprehension and mathematics problem-solving. They regarded an exclusive focus on “low-level learning” as harmful, as such a focus could result in stressing factors and practices that might unfavorably influence students’ later achievement. However, the author also agrees with Creemers and Kyriakides (2008, p. 21) that basic learning and basic knowledge are required before higher-order learning and thinking skills can be developed. In consequence, especially in primary education, a certain focus on these basic cognitive outputs is still valid in modern societies.

Taking the different main tasks of schools listed above into consideration, the extension of ed-ucation beyond the acquisition of cognitive knowledge and skills cannot be denied. Thus, social skills, problem-solving skills, and personal competences – such as responsibility and initiative-taking – are also regarded as increasingly important (Creemers & Kyriakides, 2008; Raven, 1991). Delors (1996, p. 212) for example, in his report for UNESCO, emphasized the im-portance of a good civic education “in the struggle against exclusion of all those who for socio-economic or cultural reasons find themselves marginalized in present-day societies.” Becoming

a good citizen, but more in the sense of showing loyalty to the State and its leaders – as de-scribed by Fend (2006) as the social function of the educational system to legitimize the respec-tive system of government – also was mentioned as an important quality criterion for an effec-tive schooling in the Gulf Area (Khan, 2015).

While schools likewise can contribute to non-cognitive outcomes, studies have shown that the impact of education on these domains, which are usually less prioritized in the curricula, is often rather small (Gray, 2004; Opdenakker & van Damme, 2000). Moreover, research shows that affective and cognitive outcomes do not necessarily concur. Affective outcomes here are used in the sense of Knuver and Brandsma (1993, p. 190) as the students’ attitudes towards school and learning. Their study on the relation of cognitive and affective outcomes indicated a reciprocal relationship wherein higher cognitive scores increase motivation and well-being, which in turn increase cognitive results (Knuver & Brandsma, 1993). Isac (2015, p. 139), who investigated effective citizenship education, likewise reported that for non-cognitive outcomes schools hardly would make any difference.

It is therefore argued here that, as similarly concluded by other researchers, using achievement measures in basic subjects (and consequently the approach chosen for this thesis) still has some justification in EER, especially on primary level.

The Equity Dimension

Apart from a perspective of measuring educational quality as achieving good results in certain outcome areas (described as quality or excellence), the question of the extent to which educa-tional systems are able to reduce the differences or variance between different subgroups of students, independent from their antecedent conditions, can also be asked. In many educational systems, a certain compensation for different and non-malleable context conditions, in the di-rection of more equal opportunities in the labor market, is seen as an important function of the educational system. The OECD (2012) argues that both the quality and the equity dimension need to be regarded in order to obtain a high-performing education system, providing empirical evidence using the PISA 2009 (OECD, 2010a) results. They define the equity dimension as follows: “Equity in education means that personal or social circumstances such as gender, eth-nic origin or family background, are not obstacles to achieving educational potential (fairness) and that all individuals reach at least a basic minimum level of skills (inclusion)” (OECD, 2012, p. 9). However, as discussed by Schwippert (2001, pp. 27–30), due to limitations in resources and a limited amount of time teachers will have to decide about the distribution of time and

attention to different students based not only on their motives and beliefs, but also social ex-pectations and curriculum guidelines. Consequently, teachers will have to find a balance be-tween the quality and the equity dimension when allocating their time. Heckhausen (1981) dis-tinguished different kind of allocation strategies: the need principle [Bedürftigkeitsprinzip]

where the focus of time allocation is on students showing a certain deficit with regard to an educational objective, the justness principle [Prinzip der Billigkeit] where the support is related to the achievement level of a student, and the equality principle [Gleichheitsprinzip] with equal allocation of time and attention to each student. Ultimately, the relation between these princi-ples specifies the quality criterion applied.

In the early period of school effectiveness research, the equity dimension dominated in part, and strong movements tried to investigate inequalities among different student groups (see Jencks, 1972 and Edmonds, 1979) and launch school improvement projects especially for low-SES students (‘the urban poor’). Unfortunately, results in this regard proved to be rather modest.

Teddlie and Reynolds (2000) summarized research giving evidence for the existence of differ-ential effects for schooling in terms of prior attainment, socio-economic indictors, gender, and ethnicity (known as differential effectiveness research); it seemed, however, that research re-sults were rather inconclusive and especially did not clearly indicate that more effective schools – as defined in the classical sense – would contribute to a closing of the achievement gap.

Summarizing the knowledge base, Kyriakides (2004) concluded that effective schools are able to promote learning of their students but may not have a special impact on disadvantaged stu-dents. On the other hand, there are certain rather consistent findings that “Children from disad-vantaged backgrounds are likely to be more affected by their schools than other groups across all schools” (Chapman et al., 2015, p. 96). It should be noted, however, that there is still not sufficient understanding regarding which effectiveness factors may be responsible for these differential school effects.

Summarizing the above-mentioned findings, it can be concluded that dimensions of both quality and equity should be regarded for further projects in EER.

3.3 General Effectiveness Factors