Process evaluation

The process evaluation aims at describing why an intervention works or fails. In our special case we made the experience, that our intervention worked in Germany to a certain degree, but not in Sweden. In case of a failure the process evaluation is of special interest in terms of relevance, implementation and impact of participants.

Relevance of activities to reach the goals would give us an indication of possible

the-oretical failures (good goals but not the right activities). The implementation of activi-ties by trainers was evaluated in order to find possible problems with the teaching and activities (implementation error). A final but important source of error is partici-pant error due to mistakes in e.g. matching participartici-pants’ prior knowledge with the content of the intervention. Participant error could also result from low levels of moti-vation and readiness for change in the organization (KARLSSON & VESTMAN, 2010) or giving the (right) intervention to the (wrong) people. Also external conditions could have counteracted. RANDALL und NIELSEN (2012) argue for an “environmen-tal-intervention fit”. Often a bad fit may be the case when other processes of restruc-turing are going on in the company at the time of the intervention. BIRON (2012) gives another list of contextual factors, among others a lack of resources (financial, human, expertise, skills) of the company. We start with explaining the model used in more detail.

Model and data used

The general aim of the intervention – changing leaders’ behaviour into a more re-warding and health supporting form, was divided into a number of sub-goals (see below) with the aim to clarify the objectives of the intervention in more detail for the participating leaders and team members. Once decided, these sub-goals on individ-ual, team and organizational levels were used as guidelines both for the content and the different activities in the intervention. These sub-goals for teams and leaders were evaluated during and after the implementation of the intervention. Our main fo-cus for the process evaluation in this report will be to use the process data collected to evaluate effects of the intervention for individual leaders and their teams. As a re-minder, s summary of process goals, content and activities is shown in table 6.33 below.

Tab. 6.33 Goals, content, and activities in the health promoting intervention Process goal Content Method /

process/activities Team level 1. Improved working

methods

3. feedback on leader- ship behavior

Table 6.34 below shows data used for the process evaluation. Due to budget cuts and practical reasons related to how the intervention was conducted, it was not pos-sible to conduct the evaluation for both countries in the same way. This lead to slight-ly different evaluation strategies in Germany and Sweden but efforts were made to use the same methods whenever possible. Results will be reported from the evalua-tion of Workshop I, Workshop II and the final evaluaevalua-tion after the intervenevalua-tion. The first workshop (WS I) was conducted in spring 2011 and was evaluated by all partici-pants during the workshop in Germany. In Sweden this evaluation was done a few weeks later by both leaders and employees. The second workshop was conducted almost one year later (WS II) in winter and spring of 2012. This time the evaluation by all participants was made at the end of WS II both in Germany and Sweden. The final evaluation was made after finishing the intervention in Sweden (June 2012) and as part of the T3 data collection in Germany (December 2012). After the final question-naire data collection an external researcher in Sweden interviewed 12 of the 17 par-ticipating leaders.

Tab. 6.34 Data used for the formative evaluation of the intervention

Module Germany Variables Sweden Variables Workshop I 1) short

question-naire for

Workshop II mainly like WS1 mainly like

WS1 partly same ques-tionnaire as in

items part of the

t2-questionnaire context factors: changes in the team, changes concerning the work tasks, new technical equip-ment, participation in other inter-ventions etc

items part of the t2-questionnaire

Continued Tab. 6.34

Module Germany Variables Sweden Variables total

assessment of different aspects of the training (importance of content, acitivities, clearness of goals, quality of implementation),

assessment of the eight different modules of the training

level of own engagement engagement of the team

(and some additional qualitative questions) context factors: changes in the team, changes concerning the work tasks, new technical equip-ment, participation in other interventions etc items part of the t2-questionnaire for leaders

The larger part of these data based on additional questionnaires or interviews were collected by student researchers or PhD students not paid by project funds.

In the following sections we first will give results based on teams. The first two sec-tions are about particular modules of the intervention that is WS I and WS II, based on data from participants and (partly) students observing the workshops. The third section will be to compare the most satisfied teams with the less satisfied.

The next part of our process evaluation is based on leaders’ data and is related to all modules of the intervention. We first will have a look at the evaluation we asked the leaders after the end of the intervention by questionnaires. Next we make a special analysis dividing the leaders in those who had a progress in transformational leader-ship and those who had no gain during the time of intervention. The last step will be to give results from the interviews with leaders.

Before we come to the conclusion a short section will give the overall differences be-tween Germany and Sweden in how the intervention was set into action.