Evaluation of Workshop I

Our main interest for the first workshop was to find out, if participants were satisfied with the workshop, because it is argued that statisfaction contributes to positive

out-comes of the intervention (MURTA, SANDERSON & OLDENBURG, 2007) and which characteristics and elements of WS I contributed to the satisfaction level.

Germany

Participants

In Germany, 11 teams were recruited to take part in the intervention, with overall N = 87 participants. Nine of the teams work in municipalities, two teams work for a private bank (see Chapter on summative evaluation for further information regarding the German and Swedish samples).

Measures

As predictors we evaluated a set of WS characteristics to be grouped into 1) context variables, 2) process variables 3) team characteristics: resources and problems (based on t1 data, compared to the mean of the other groups) and 4) content of the WS I: impact of feed-back about t1 data and quality of action plans.

As context variables we included time pressure during the workshop and team size.

Process variables were omitted program parts, deviations from the agenda, protest of participants toward elements of the workshop, questions of participants about pre-measurement and additional agreements with participants after the WS.

In general there were two data resources for the process evaluation: Each participant filled out a two-page questionnaire at the end of the workshop. Additionally, a trained student assistant accompanying the trainer had to fill out a questionnaire with a se-ries of questions describing the above mentioned context and process variables.

Problems and resources as team characteristics were measured in the following way:

If the group in question was significantly below average for a positive construct of the T1 questionnaire (e.g. Role clarity) or significantly above average for a negative con-struct (e.g. Workload), this would count as one problem. Otherwise, if it was above average for positive or below average for negative constructs, it was seen as a re-source. All the team problems were then accumulated to a team problem score. The same was done with resources.

Potential predictors for the success of a workshop

In this report we will only present a choice of criteria that have been measured.

Impact of T1. The subjective impact of the presentation of the results derived from the first survey (T1) is of special interest for this evaluation because it is strongly linked to the participants’ expectations of the workshop (participants have reported strong interest in feedback about the current status of their team during the realisa-tion of the first survey). It was measured via a self-constructed scale of seven items, again with a five-point Likert scale. The participants were asked to what extent they perceive the results of the workshop as novel, and interesting. The scale yielded a Cronbach’s alpha of .83.

Quality of action plans. The quality of the action plans, which were developed as an important part of the workshop, can be seen as an indicator for possible skills that the participants could have acquired during the workshop. To measure this item, a scor-ing system was developed to assess the complexity of each plan. The action plans consisted of seven levels (targets, sub-targets, procedure, dates, responsibilities, obstacles and suggestions for coping, see figure 6.2). The plans become more and more sophisticated with each level (DE BONO, 1992). The scoring system was sim-ple: For every given answer on every level one point was added. Since some groups had more targets than others, the resulting sum of points was divided by the number of goals derived.

Group size. The size of each workshop group was taken from the total number of evaluation questionnaires completed by its members. So, only participants that were present at the end of the workshop session counted, regardless of whether they were present from the beginning or not.

Problems and Resources. Both Variables can be seen as a kind of baseline of the workshop groups’ individual status quo, derived from the first measurement T1: If the group in question was significantly below average for a positive construct of the T1 questionnaire (e.g. Role clarity) or significantly above average for a negative con-struct (e.g. Workload), this would count as one problem. Otherwise, if it was above average for positive or below average for negative constructs, it was seen as a re-source. All the team problems were then accumulated to a team problem score. The same was done with resources.

Team climate. Unfortunately, the team climate was only described by a short, per-sonal review of a person who observed the workshop. To derive useful data from these information, a group of three experts rated each description for the three self-created sub facets communicativeness, equality of the share of speech and atmos-phere (tense vs. relaxed). If a sub facet did not seem to fit to the observer’s review, it was omitted for the corresponding group. Each one of the experts rated the team climate for every workshop. Then the sub facets were combined by calculating the arithmetic mean. For each team, three team climate ratings resulted. These ratings were tested for inter-rater reliability by a two-way mixed model intraclass correlation, resulting in a very good ICC of .87. The mean of the three ratings for each group was used as team climate score.

Outcome variable

Satisfaction. The subjective satisfaction of participants with the workshop can be seen as a very useful outcome variable, for it is highly related to factors of high im-portance for a successful transfer of workshop contents, like motivation (VOHS &

BAUMEISTER, 2008) or compliance (SMITH, LEY, SEALE, & SHAW, 1987). Satis-faction was assessed with a self-constructed scale using ten items and a five-point Likert scale. The participants were asked, among other things, the extent to which they found the workshop to be useful, interesting, applicable, and how much they were likely to recommend it to others. The scale yielded a Cronbach’s alpha of .93.

Results

Figure 6.23 gives an overview of satisfaction ratings across items. Satisfaction with the workshop was overall good (M = 4.13, SD = 0.72, n = 87), with the majority of participants giving ratings on ”agree” and ”completely agree” to all items. Especially ratings related to the structure, and atmosphere in the workshop were very positive.

A potential transfer of workshop contents to daily work was seen slightly less favora-bly. However, the majority of participants (59.1 %) perceived that the workshop will positively change their work, or that they will be able to transfer contents from the workshop to their daily work (65.1 %).

Fig. 6.23 Satisfaction with the workshop in Germany (N = 87)

Our expectation was that context and process variables are of less importance than the content of our intervention. Concerning the characteristic of teams – teams that are better off and those who are worse off – we hoped that our intervention will satis-fy both groups.

Using HLM we found out, that greater team size and having more problems com-pared to other teams were related with lower satisfaction whereas perceiving an im-pact of results-feedback from the t1 data collection wave was related with higher sat-isfaction.

Obviously feedback about the current state of the team seems to be the most im-portant feature for the satisfaction with the workshop. Feedback on survey results can be seen as a standalone intervention.

0,0% 20,0% 40,0% 60,0% 80,0% 100,0%

This workshop is…

We were pleased to find out – after controlling for additional variables – that there were no differences due to the trainers. This leads us to assume that our trainers had comparable and adequate level of competencies in order to handle the different teams.

Sweden

Participants

In Sweden, 16 teams were recruited to take part in the intervention, with overall N = 238 team members and 16 leaders at the start. All teams worked in two municipali-ties.

Measures

Data collection and measures for WS I differed from those of Germany due to con-straints of capacities. Whereas in Germany leaders and their team members com-pleted a questionnaire immediately after the intervention, in Sweden the data collec-tion with leaders and team members was done on two distinct dates. Leaders got a questionnaire few month after the end of WS I with 15 questions (see figure 6.25) and team members were asked about WS I at WS II with three questions (see figure 6.26) almost one year later.

Results from leaders

The 16 responding leaders in Sweden were generally positive in their evaluations of content and relevance of the workshop and slightly more positive towards the work-shop as a whole (M = 3.70, SD = 0.56 compared to M = 3.45, SD=0,58, n = 16) for the evaluation of the questionnaire feedback. Lowest value concerned whether any-thing learned from the workshop had an influence on their work as leaders and if it changed anything in the workplace.

Fig. 6.24 Evaluation of WS I by Swedish leaders (n=16), scale 1-5 (Completely Disagree to Agree completely)

Results indicate that the Swedish leaders in general were quite satisfied with the first workshop and especially with the feedback on their own team’s responses to the first questionnaire. Although they gained new knowledge about how their team members perceived working conditions and their leadership, the practical implications of this for their daily work as a leader seemed to be less obvious.

Results from teams

The items used the past tense to ask about how WS I was perceived. Responses were given on a five-point scale ranging from “not at all” through “partly” to “To a very high degree”. Per cent and mean values of the three items are shown below:

Fig. 6.25 Evaluation of WS I in Sweden

Most positive perceptions were given on the first item regarding the usefulness of WS I, i.e. the questionnaire feedback (T1) and action plans made.