9 RESULTS OF VARIABLE SELECTION AND FACTOR/ RELIABILITY ANALYSES
9.4 Results of the Correlation Analyses
9.4.2 Course and school level
The following section presents the results of the correlation analyses between all level 2 varia-bles components that were retained from the previous analyses steps. Correlations with mathe-matics achievement are shown in Table 9-30, and the corresponding results for science achieve-ment in Table 9-31. Level 2 correlations were calculated by correlating the course averages of the predictor variables with the course averages of the achievement scores, using the IEA IDB Analyzer. Corresponding sample sizes can be found in Table 7-1. Statistically significant asso-ciations are marked in bold.
For group-level variables, a check for which variables would not show a relation of at least 0.2, in any of the countries for any of the subjects, was administered. Such variables were then removed from further analysis steps – unless the literature review indicated a special importance of that specific component. Correlations partly differ across subjects, as can be seen when com-paring the results of both tables. Altogether, the following nine variables were identified as having low positive correlations with achievement for both subjects in all countries (with some of them having unexpected higher negative correlations):
Table 9-30: Correlation of course and school level factors with mathematics achievement
Notes. significant correlations (0.05 level (2-tailed)) are marked in bold
BHR = Bahrain, KWT = Kuwait, OMN = Oman, QAT = Qatar, SAU = Saudi Arabia, ARE = United Arab Emirates
Number of computers in school JCBG11 0.00 -0.04 0.00 0.04 0.09 0.25 Availability of school library and # of books JCBG13A 0.25 0.35 0.00 0.13 0.11 0.46 Shortage of resources F_SHORTAGE_M 0.20 0.07 0.03 0.30-0.02 0.29 Emphasis on academic succes F_SC_EAS 0.23 0.34 -0.01 0.30 0.23 0.41 School discipline and safety F_SC_SOS 0.25 0.17 -0.09 0.08 0.06 0.30
Instructional time JCDG08HY 0.08 0.14 0.05 -0.14-0.09 0.06
Problems with absenteeism F_ABSENCE 0.19 0.22 0.02 0.14 0.11 0.33
Opportunity Policies related to tracking JCBG10A -0.18-0.05 0.11-0.08 -0.04 -0.01
Teaching experience (years) JTBG01 0.14 0.14 0.07 0.08 -0.11 0.02
Gender of teacher JTBG02 0.08 -0.05 0.09 -0.11 0.30 -0.01
Teacher's highest education level JTBG04 -0.08 0.07 -0.26-0.01 0.01 0.04 Teacher majored in edu. and subject JTDM05 0.08 -0.04 0.05 0.07 0.00 -0.08 Time spent on professional development JTBM10 -0.02 0.04 -0.01 -0.20 0.02 -0.15 Confidence in teaching F_CONFIDENCE_M 0.11 0.04 0.09 -0.10 0.25 0.16 Preparedness to teach subject JTBM11Z 0.12 0.16 0.11 0.11 0.02 0.08 Average economic and sociocultural status F_ESCS_clX 0.55 0.63 0.17 0.76 0.22 0.76 Average early numeracy skills F_EARLYNUM_clX 0.14 0.55 0.14 0.41 0.08 0.37 Average gender composition JTSEX_clX 0.23 0.13 0.09 0.04 0.37 0.04 Average composition in terms of non-nationals JNATIONAL_clX -0.02 0.53 -0.18 0.60 0.14 0.59 Structured teaching Clear and structured teaching F_CLEARST_M 0.25 0.17 0.34 0.46 0.38 0.41
Activation Cognitive activation F_COGNACTIV 0.20 0.10 0.02 0.01 0.16 0.26
Management Limitation of teaching (disruptive students) JTBG15D 0.14 0.21 -0.03 0.21 0.25 0.27 Emphasis on academic success F_TCH_EAS 0.19 0.15 0.21 -0.04 0.26 0.26 Orderly learning environment F_ENVIRONM 0.27 0.09 0.09 0.14 0.24 0.39 Verification of homework assignment JTBM07Z 0.05 0.01 0.10 0.03 0.26 0.17
Monitoring progress JTBM08A 0.08 0.05 0.02 -0.01 0.14 0.10
Teaching time spent on subject JTBM01 0.06 0.26 0.01 -0.02 0.11 0.13 Amount of homework assigned JTDM07Z -0.10 0.22 -0.07 -0.12 -0.02 0.12
Opportunity Number of topics taught JTDM06Z 0.13-0.03 0.18 0.10 0.14 0.01
Course
Input
Teacher background
Student composition
Quality of
Instruction Climate Assessment Time
School
Input Resources
Quality Environment (SLE) Time
AV
Level Factor Factor - Details Variable Description Variable BHR KWT OMN QAT SAU ARE
Table 9-31: Correlation of course and school level factors with science achievement
Notes. significant correlations (0.05 level (2-tailed)) are marked in bold
BHR = Bahrain, KWT = Kuwait, OMN = Oman, QAT = Qatar, SAU = Saudi Arabia, ARE = United Arab Emirates
Overall instructional time (school level)
The overall instructional time was derived from three questions related to the number of days the school is open for instruction, the typical instructional time per day, and the number of days per week. Sufficient instructional time is needed for teaching a certain course syllabus to the students; thus, an association between the total instructional time available and achievement outcomes could be assumed. As the amount of instructional time devoted to core subjects in the Gulf area used to be relatively low in international comparisons, in the last years, Gulf countries reacted and enacted reforms allotting more time for mathematics and science, such as the daily school timing initiative in Bahrain (Al-Awadhi, 2016, p. 8). Concerning the finding that corre-lations between overall instructional time on school level and achievement don’t show the ex-pected higher correlation results, one reason could be that the amount of official school days, and their length, might be prescribed on national level – and hence, not too many differences between schools occur. Additionally, the question in the TIMSS questionnaire only relates to the typical school day. School closures, or periods without formal instruction due to natural disasters, strikes, or longer school closures during national testing periods or festivities and so forth, are not explicitly covered in the questionnaire.
Number of computers in school JCBG11 -0.02 -0.04 0.03 0.06 0.06 0.28 Availability of school library and # of books JCBG13A 0.21 0.30 -0.01 0.13 0.10 0.48 Shortage of resources F_SHORTAGE_S 0.19 0.00 0.01 0.22 0.01 0.24 Emphasis on academic succes F_SC_EAS 0.21 0.34-0.03 0.34 0.27 0.42 School discipline and safety F_SC_SOS 0.31 0.17 -0.09 0.09 0.08 0.32
Instructional time JCDG08HY 0.05 0.07 0.01 -0.10 -0.05 0.04
Problems with absenteeism F_ABSENCE 0.23 0.15 0.02 0.14 0.10 0.34
Opportunity Policies related to tracking JCBG10B -0.08 -0.04 0.11 -0.18-0.10-0.11
Teaching experience (years) JTBG01 0.05 0.16 0.06 0.03 -0.19-0.07
Gender of teacher JTBG02 0.12 0.11 0.08 -0.05 0.54 0.05
Teacher's highest education level JTBG04 -0.08 0.20-0.15 -0.01 0.04 0.10 Teacher majored in edu. and subject JTDS05 0.14-0.05 0.04 0.14 -0.13 -0.03 Time spent on professional development JTBS09 -0.02 -0.05 -0.04 -0.19-0.02 -0.09 Confidence in teaching F_CONFIDENCE_S 0.05 0.14 0.04 -0.07 0.13 0.20 Preparedness to teach subject JTBS10Z 0.14-0.11 0.11 0.10 0.14-0.09 Average economic and sociocultural status F_ESCS_clX 0.40 0.59 0.15 0.70 0.32 0.78 Average early numeracy skills F_EARLYNUM_clX 0.12 0.41 0.18 0.43 0.15 0.42 Average gender composition JTSEX_clX 0.34 0.23 0.15 0.18 0.55 0.08 Average composition in terms of non-nationals JNATIONAL_clX -0.01 0.43 -0.16 0.57 0.18 0.65 Structured teaching Clear and structured teaching F_CLEARST_S 0.30 0.24 0.37 0.51 0.40 0.46
Activation Cognitive activation F_COGNACTIV 0.13 0.18 0.07 0.16 0.26 0.30
Management Limitation of teaching (disruptive students) JTBG15D 0.21 0.17 0.02 0.21 0.12 0.31 Emphasis on academic success F_TCH_EAS 0.05 0.20 0.14 0.09 0.34 0.30 Orderly learning environment F_ENVIRONM 0.24 0.15 0.06 0.22 0.20 0.40 Verification of homework assignment JTBS06Z 0.08 0.14 0.00-0.29 0.23 -0.01
Monitoring progress JTBS07A 0.00 0.06 0.14 0.03 0.00 0.05
Teaching time spent on subject JTBS01B -0.04 0.20 0.11-0.17 0.11 -0.04 Amount of homework assigned JTDS06Z -0.01 0.09 0.00-0.17 0.14 -0.03
Opportunity Number of topics taught JTDS05Z 0.17 -0.22 0.03 -0.02 0.01 -0.03
Course
Input
Teacher background
Student composition
Quality of
Instruction Climate Assessment Time
School
Input Resources
Quality Environment (SLE) Time
Level Factor Factor - Details Variable Description Variable BHR KWT OMN QAT SAU ARE
Tracking policy according to mathematics and science achievement, respectively
There are certain indications that tracking and streaming might influence learning opportunities of the students (see section 3.3.3), but the literature does not give clear empirical evidence for a straightforward association between tracking procedures and achievement for different stu-dent subgroups. The general “yes/no” question asked in the TIMSS questionnaire is likely not specific enough to delve deeper into this issue, and therefore doesn’t allow for the explanation of possible influences of tracking policies on student achievement.
Teachers’ experience in years
Findings in relation to the mere gains in student learning measured by teachers’ teaching expe-riences in years seem to be somewhat mixed. While TIMSS trend analyses over all countries show that achievement was highest, especially for mathematics, for teachers with more than 20 years of experience in grade eight (Mullis, Martin, Foy, & Arora, 2012, p. 292), other authors could not confirm a relation or assumed a rather curvilinear effect. Associations between teacher background factors and student achievement are further discussed in section 3.3.5.1.
Teachers’ highest education level
Interestingly, in some of the countries, the teachers’ highest education level is negatively asso-ciated with achievement – for example, with a correlation coefficient of -0.26 for mathematics in Oman. On the contrary, analysis from Blömeke et al. (2016) indicated the teacher educational level as the strongest predictor for student achievement across the TIMSS 2011 countries. Be-cause of the interesting and contradictory nature of relations to achievement within the GCC countries, and the relative importance of the variable in other research, it was decided to keep the variable for the multilevel analyses.
Teachers’ specialization in math and education
There is some indication in the research literature that the specialization and formal education of the teachers is related to student outcomes. However, the index created by the TIMSS &
PIRLS International Study Center, which stems from two questions related to teacher’s formal post-secondary education and their main area of specialization (TQ-G5A/B) only showed low correlations with achievement in all GCC countries – and therefore was dropped from further analyses.
Time spent for professional development
Based on the comparatively low quality of education in the Gulf area, and on research findings similar to Blömeke et al. (2016), who found based on TIMSS 2011 data that professional de-velopment activities are especially important for the Arab countries, it could be assumed that a higher amount of time spent for professional development might be related to students’ mathe-matics and science achievement. However, in addition to the amount of time spent for training, factors like the quality and content coverage of the courses also play an important role. In this context, correlations to student achievement using TIMSS 2015 data are partly negative, reach-ing -.20 in Qatar. Because of the importance attributed to professional development by other researchers, the variable was kept to be evaluated further in the multilevel analyses.
Preparedness to teach
An indicator summarizing several variables related to teachers’ perceived preparedness to teach, as related to different content domains, was created to give an indication about subject matter mastery. This can be seen as a basic requirement for good teaching, as indicated for example by Monk (1994). However, related literature revealed less consistent and rather weak relations to student achievement, as further described in section 3.3.5.1. Given the weak corre-lations with achievement in the region, the indicator ultimately was dropped from further anal-ysis steps.
Assessment of ongoing work in mathematics and science
The emphasis on assessment of student’s ongoing work was selected as one component for the assessment dimension of the factor quality of instruction. There is quite some empirical research evidence for the importance of evaluating students’ work and giving timely feedback that can be used for student’s improvement. However, the question, as asked in the TIMSS 2015 ques-tionnaire, didn’t collect information regarding how the information is going to be used – i.e., whether it was summative (as a kind of final judgment) or formative (which would mean that the results are used to influence subsequent teaching and learning strategies). The latter con-struct was especially found to be more strongly related to student performance (see assessment and feedback strategies in section 3.3.5.2). Because of the low correlations, this variable was excluded from further analyses.
Number of topics covered for math/science
The curriculum content coverage is usually regarded as an important indicator for students’
opportunity to learn; correspondingly, a question asking teachers about the curriculum coverage of the TIMSS topics is included in all cycles of TIMSS. Because of its theoretical importance, this variable will be kept for further analyses, in spite of low correlations indicated for the GCC countries. Interestingly, for Kuwait, the science content coverage is even significantly nega-tively associated with student achievement.
In a subsequent step, and based on the current research framework, the extent to which selected variables and indicators represent the same or a very similar construct was investigated. In such cases, the indicator with the strongest correlation to mathematics and science achievement was kept, and the remaining indicators for the same construct were excluded from further analyses.
As the starting point here was a regional analysis, certain compromises had to be made. In order to maintain the same variable set for all countries, the average correlation was considered for the selection process; additionally, the extent to which countries differed in their associations between indicators and achievement was investigated. In some cases, constructs between course and school level were also parallel. In detail, the following components were regarded.
Educational resources on school level
In total, three explanatory variables (the number of computers, the number of books in the school library, and the principal’s perspective on how much the instruction is affected by spe-cific shortages) were available in the principal questionnaire as possible indicators for educa-tional resources. While nearly no correlations between resource indicators and achievement could be found in Oman, for all other countries, except Qatar, the strongest indicator by far was the number of books in the school library. For Qatar, the strongest indicator was the principal’s perspectives on how much the instruction is affected by shortages. Here it was decided to drop the weakest component, namely the number of computers available for fourth grade students, and to keep the other two.
Learning environment
The school questionnaire contained two questions related to the school learning environment, one related to the emphasis on academic success and another to school discipline and safety.
While both are important determinants for the school climate, they nonetheless likely to be related, to a certain extent. To keep the model parsimonious, it was decided to keep emphasis
on academic success, which showed, by far, a stronger correlation to both mathematics as well as science achievement in five of the six countries.
The question about emphasis on academic success was also administered to the teachers, in addition to a question related to the orderly learning environment. As the latter had a stronger correlation to achievement on course level, it was kept for the multilevel analyses as an indicator for a supportive climate on course level.