Data reduction procedures

This section describes the steps undertaken to reduce the number of variables to a smaller num-ber of underlying factors that could be used in the subsequent multilevel analyses. In order to construct valid scales, sets of items were first inspected by principal component analysis (PCA), which belongs, in a wider sense, to the methods of factor analyses. Once a meaningful and statistically sound factor solution was obtained, the internal consistency of the items contrib-uting to the final scales were examined through reliability analyses. Construct-validity of the final scales, as well as of single items that were retained for further analyses, were examined by way of correlation analyses, which investigated the relationship between the retained scale or variable and mathematics and science achievement. Principal component and reliability anal-ysis were conducted by the statistical package SPSS^TM (IBM Corp., 2011) and applied in par-allel to all five imputed datasets. For this analysis step, a regional approach was followed, mean-ing that all data from the six GCC countries were pooled together.

8.3.6.1 Principal component analysis

Factor analysis in general allows for the combination of variables with common characteristics.

Two general methods can be distinguished: exploratory factor analysis and confirmatory factor analysis. While the former, especially in its form as principal component analysis, allows for the identification of underlying patterns in previously unknown groupings of variables, con-firmatory factor analysis is used to verify a hypothesized relation and grouping of certain iden-tified factors (Cohen et al., 2007, p. 560). As the objective for the current study was to specify the underlying constructs in the region, principal component analysis was applied to extract the main factors in the form of a regional approach. This means that the analyses were conducted

on the pooled data of the six GCC countries by weighting each country equally. As no teacher questionnaire data was available for up to around 6% of the teachers in the six countries, sepa-rate group-level sampling weights were calculated to assure that each country has exactly the same weight of 500. For general variables, the adjusted group-level weights were based on the TIMSS general teacher weight (TCHWGT), while mathematics-related variables were based on the mathematics teacher weights (MATWGT), and science-related variables on the science teacher weights (SCIWGT). The analyses were applied to each of the five imputed data sets separately, based on the variables identified as being related to the conceptual framework in chapter 6. As only participating students with properly adjusted weights were included in the final international TIMSS database, level 1 student- (and parent-) level principal component analyses could be weighted using the available TIMSS student-level senate weight (SENWGT).

For the principal component analyses, only questions on an interval scale were included. The following steps were applied:

• The suitability of each set of variables for principal component analysis was checked by verifying the sample sizes and the item correlation matrix. For this purpose, the Kaiser-Meyer-Olkin (KMO) criterion was checked. The KMO ranges from 0 to 1 and the coefficient should obtain an absolute minimum of 0.5 (Bühner, 2011, p. 347). In addition, the result of Bartlett’s test of sphericity was considered. This test checks the null hypothesis that the sample is originating from a population where the considered variables are uncorrelated. Bartlett’s test should significantly reject the null hypothesis with p < .05 (Backhaus, 2011, p. 341).

• As the objective of this step was the data reduction towards underlying constructs, fac-tor extraction was performed by applying principal component analysis (PCA). PCA can be used to reduce a large set of possibly correlated variables to a smaller number of uncorrelated factors (the principal components). Hereby, the first factor (or linear combination of variables) is extracted, such that the maximum shared variance of the original data set can be explained. In subsequent steps, other factors are then succes-sively extracted, each time trying to explain the maximum portion of the remaining variance. The extraction results in a set of uncorrelated factors. To determine the max-imum numbers of factors to be extracted, in general the Kaiser criterion was applied, meaning that all factors with an eigenvalue larger than 1 (which would correspond to the variance of one standardized variable) are extracted. The eigenvalue is a measure for the contribution of explained variance from one single factor regarding the vari-ance of the whole variable set. Additionally, the graphical representation of the Scree

test was evaluated, and only factors above the elbow (or break) in the plot were re-tained (see Backhaus, 2011, p. 359).

• In order to allow for a better interpretation of the obtained factor results, a factor rota-tion was performed. For this study, Varimax rotarota-tion, an orthogonal rotarota-tion proce-dure, was applied. Varimax rotation maximizes the variance between factors and thus helps to more clearly identify the groups of variables that are closely correlated, and distinguish them from other variables (Cohen et al., 2007, p. 566).

• Factor loadings, which represent the correlation between original variables and ob-tained factors, were examined. Factor loadings can assume values between -1.0 and 1.0, with higher absolute values indicating stronger relationships. For the purpose of this study, loadings above 0.3 were considered as acceptable (Bühner, 2011, p. 350).

In a few cases, items with double-loadings, which means that they load on more than one factor, were removed from the model. Additionally, communalities of each item after factor extraction were checked. The explained variance of each items by the fac-tor solution should attain at least 10% (Bühner, 2011, p. 358).

8.3.6.2 Reliability Analyses

Once suitable and interpretable constructs were obtained through the PCA step, the internal reliability of the constructed scales were assessed by means of reliability analyses. The study adopted the Cronbach’s alpha (α) as a measure of internal consistency of the scales created.

Cronbach’s alpha is used for multi-item scales, and checks inter-item correlations by measuring the correlation of each item with the sum of all other items (Cohen et al., 2007, p. 507). The coefficient ranges from 0 to 1; coefficients above 0.67 or even 0.8 are mostly regarded as ac-ceptable (Cohen et al., 2007, p. 506). However, as the current study follows an exploratory research design and uses rather unreliable background questionnaire data, a lower coefficient of 0.5 for scales that are well justified from a theoretical perspective will still be considered for inclusion into further analyses. This approach is in line with other researchers, such as Bos (2002) and Cho (2010). In general, an alpha value between 0.7 and 0.8 will be judged as “ac-ceptable”, between 0.8 and 0.9 as “good”, and above, 0.9 as “excellent”. As an additional means of checking the scale homogeneity, each single item was checked to ascertain whether the whole scale would obtain a higher reliability if the item were dropped. In the case that removal of an item would enhance the scale reliability, this item was dropped from the scale.

Once factor and reliability analyses confirmed the consistency of the created scales, factor scores were saved and retained for further analyses.

8.3.6.3 Correlation Analyses

As a final step of the data reduction process, bivariate correlation analysis were performed in order to determine the association between background scales and mathematics and science achievement. First, correlations between the obtained scales, as well as single variables that were retained for further analyses, were verified to ensure that no multicollinearity occur in the data. Multicollinearity may occur when predictors are highly correlated with other predictors in the model. In multiple regressions (as will be applied for the multilevel analysis), this can interfere with determining the precise effect of each predictor in the final model. In cases of high correlations between variables, it therefore is suggested to either remove one of the con-cerned predictors or to combine the highly correlating variables into a new one (Cohen et al., 2007). For the current analyses, inter-item correlations between indicator variables higher than 0.8 (Field, 2004, p. 132) were further investigated.

For this study, the Pearson product moment coefficient was calculated with help of the IEA IDB Analyzer (see 8.3.1). The coefficients range from -1 to 1, indicating direction and strength of the relationship. Usually, correlations below 0.35 are classified as “low”, between 0.35 and 0.65 as “medium”, and above 0.65 as “high”. However, this study will apply a minimum value of 0.2 for the correlation coefficient. This low cut-off point was chosen as in exploratory studies, when considering the high sample size, it might be worthy to also explore low correlations in exploratory relationship research (Cohen et al., 2007, p. 536). A correlation could also be de-scribed as the common variance that is obtained by squaring the correlation results (Cohen et al., 2007, p. 536). This means that a correlation of 0.2 then would explain 4% of the shared variance. Those percentages of explained variances that are relatively low, nevertheless, might still be important from a policy perspective (Teddlie & Reynolds, 2000, p. 98). All level 1 cor-relations were calculated based on the country-specific student sample sizes listed in Table 7-1, while level 2 correlations were calculated between the course averages of the predictor variables and the corresponding course averages of the outcome scores.

8.4 Creating an Index of Economic Social and Cultural Status (ESCS)