Development of the TMIC-SJT

Chapter 4: Development and Validation of the Test to Measure Intercultural Competence (TMIC)

Chapter 4 describes how the TMIC-SJT was developed and summarizes the method and results of two studies that were conducted to validate the TMIC. The content is based on Schnabel et al.’s (2015) research article, which contains additional details of both studies.

4.1 Development of the TMIC-SJT

First, I took the 17 self-appraisal scales from the TMIC (TMIC-SA) as inferred from the pretest and used them as a starting point. Then, I developed the situational judgment items to complement the TMIC-SA.

The construction of the TMIC-SJT was based on the approach recommended by McDaniel and Whetzel (2005). Twenty critical incidents (Flanagan, 1954) developed by expat managers, expatriates, and intercultural trainers served as the basis for the items. They included a typical international cooperation situation as well as statements about how an individual would ideally behave in this situation and which behaviors would possibly make the situation worse. The critical incidents were job- or country-specific in part and therefore had to be generalized. They were reformulated

accordingly and converted into a suitable format. The situational judgment items that were designed were then assigned to the 17 factors in a double-blind procedure with six intercultural experts.

Text-based situational judgments and behavior-related answer alternatives were chosen (“How are you most likely to behave in this situation?”) as opposed to knowledge-related (“What is the best answer?”) answer alternatives, as the former have a higher correlation with behavioral variables (McDaniel, Hartman, & Grubb, 2003; McDaniel et al., 2006), are more culturally independent (Nguyen, McDaniel, & Whetzel, 2005), and correspond more with the specific aim of showing the behavioral aspect of ICC. An example of a situational judgment item measuring flexibility in

communication is given in the following; all situational judgment items are available as supplemental material:

You are to pass an important message on to a foreign colleague. However, the person’s reaction shows that he or she has clearly not understood it. How are you most likely to behave?

!

a) I will leave it to someone else to pass on the message.

b) I will repeat the message using the same words but will speak more slowly and louder so that the person I am speaking to can understand me better.

c) I will repeat the message and pay close attention to my choice of words.

d) I will ask my colleague what he or she did not understand and reformulate the statement accordingly.

The TMIC-SJT instructions explicitly invite the test participants to choose one answer out of four. This procedure is commonly found on situational judgment tests (McDaniel & Whetzel, 2005).

4.2 Study 1: Factor Structure, Psychometric Properties, and Criterion Validity of the TMIC The first study involved N = 641 German students. Seventy percent were female (488) and 30%

(193) were male. The study participants were 28.25 years old (SD = 9.29) on average.

In order to investigate whether the 17 scales that were inferred from the pretest fit the data well, exploratory structural equation modeling (ESEM; Asparouhov & Muthén, 2009) in Mplus (Muthén &

Muthén, 1998−2012) was used. The following changes were made on the basis of the ESEM results:

Four items were excluded due to loadings < .20 (Bortz, 2006). Nine items were assigned to factors other than those that were originally anticipated. One factor (specific use of diversity) was omitted and another factor was split into two factors (integration in groups). The final structural equation model consisted of 75 self-appraisal items and was tested for its goodness of fit, which was found to be highly satisfactory, χ²(1636, N = 641) = 2579.85, p < .001; χ²/df = 1.58; RMSEA = .031, 90% CI [=

.029, .033]; SRMR = .017; CFI = .96; TLI = .93. The numbers and examples of items on each factor are summarized in Table 2.

In a second step, a model containing the situational judgment items was tested. As a result of the aforementioned departures from the pretest model, situational judgment items were available for only 16 factors at this stage. Thus, the model fit was assessed for 17 first-order factors with 75 self-appraisal items and 16 situational judgment items. Furthermore, six second-order factors were included to explain the 17 first-order factors. As second-order models could not be computed with ESEM in Mplus (Asparouhov & Muthén, 2009), a confirmatory factor analysis (CFA) was applied.

!

This led to an acceptable model fit, χ²(3987, N = 641) = 8280.09, p < .001; χ²/df = 2.08; RMSEA = .040, 90% CI [.040, .046]; SRMR = .076; CFI = .82; TLI = .81. The internal consistency (Cronbach’s α) of the 17 factors ranged from .69 to .90. The overall reliability of the TMIC-SA was excellent (α =

.96).

Furthermore, criterion validity was established with the help of three external criteria: (a) duration of previous stays abroad, (b) previous participation in an intercultural training, and (c) private or professional intercultural involvement. Because a latent general factor was theoretically and statistically rejected, the variation in the overall TMIC value was analyzed on a manifest level. Two analyses of variance each revealed a significant effect of the influence of the length of a stay abroad on the overall value on the TMIC-SA, F(3, 637) = 36.51, p < .001, η² = .15, and on the TMIC-SJT, F(3, 637) = 9.45, p < .001, η² = .10 (for the related contrasts, see Schnabel et al., 2015). A t test showed that participants who had already participated in an intercultural training scored, on the whole, higher on the TMIC-SA, M = 4.37, SD = 0.45, as well as on the TMIC-SJT, M = 3.07, SD = 0.28, than people who had never participated in an intercultural training, M = 4.14, SD = 0.48; t(639)

= 7.87, p < .001, d = 0.69, and, M = 2.94, SD = 0.31; t(639) = 5.11, p < .001, d = 0.40, respectively.

Another t test revealed that professionally or privately interculturally involved individuals achieved a higher overall value on the TMIC-SA, M = 4.31, SD = 0.47, than participants who said they were not interculturally involved, M = 3.90, SD = 0.44; t(639) = 8.69, p < .001, d = 0.70. This also applied to the TMIC-SJT, t(639) = 5.27, p < .001, d = 0.52, such that individuals who had affirmed their intercultural involvement had greater TMIC-SJT scores, M = 3.01, SD = 0.30, than those who said they were not interculturally involved, M = 2.85, SD = 0.33. In addition, multiple group comparisons with a Bonferroni-Holm-corrected significance level of p < .001 (Holm, 1979) were conducted with sequential equation modeling (SEM) for the 17 latent factors and the external criteria (a) previous participation in intercultural training and (b) private or professional intercultural involvement.

Previously interculturally trained and interculturally involved individuals scored higher on 13 of the 17 latent factors. All results are extensively summarized in Paper A (Schnabel et al., 2015).

!

To further validate the TMIC, correlations between the TMIC-SA and TMIC-SJT, as two different methods, were computed for all first-order factors. This led to one nonsignificant correlation, r(641) = .05, ns (flexibility in communication), one significant yet very low correlation, r(641) = .08, p < .05, and 15 other significant correlations that ranged from r(641) = .11, p < .05 (sensitivity in communication) to r(641) = .61, p < .001 (willingness to use a foreign language). The overall TMIC-SA was moderately correlated with the overall TMIC-SJT, r(641) = .49, p < .001. This corresponds to previous findings (cf. Bledow & Frese, 2009) that showed that although the two methods measure the same construct, the SJT measures behavioral preferences, whereas the report scale captures self-concept. In addition, the incremental validity of the TMIC-SJT was investigated via regression analyses. Specifically, I examined whether the TMIC would explain the external criteria better when the TMIC-SJT total score was added to the regression model in comparison with the TMIC-SA total score alone. The results showed that adding the TMIC-SJT explained more variance in (a) duration of previous stays abroad, ΔR² = .001; χ²(1) = 22.98, p < .001, (b) previous participation in an

intercultural training, ΔR² = .009; χ²(1) = 19.10, p < .001, and (c) private or professional intercultural involvement, ΔR² = .001; χ²(1) = 20.38, p < .001, than the TMIC-SA alone. However, even if the ΔR² -values were significant, their size was very modest.

4.3 Study 2: Replication of the Factor Structure with Professionals and Construct Validation