SECTION THREE METHODOLOGY
3.1 OVERVIEW: 'REALIST EVALUATION'
The methodology for this evaluation, and the companion evaluation of the Professional Development initiatives (Peddie et al., 2003), is adapted from the
‘realist evaluation for evidence-based practice’ developed by British evaluators whose initial focus areas were often new programmes in social work (Pawson &
Tilley, 1997; Kazi, 2000; Kazi & Spurling, 2000). This approach draws from the earlier work of Michael Scriven (1994), with his notions of the three ‘boxes’, black, grey and white.
Scriven argued that evaluations which dealt only with outcomes treated the
processes which led to those outcomes as a 'black box', not open to scrutiny, and presumed to be unimportant. While this allows policy makers to assess success or failure, it offers very little by way of constructive suggestions in the latter case.
The “grey box” approach attempts to examine process, at least in part. A “white box” approach was seen as the ideal. This would be an evaluation which
holistically examines process and outcomes, and the relationships between them (Scriven, 1994).
Realist evaluation builds from this last approach, adding the notion of “contexts in practice” (Kazi & Spurling, 2000). In terms of social work it has been defined as addressing “all significant variables involved in…practice, through a realist
effectiveness cycle which links the models of intervention with the circumstances in which practice takes place.” (Kazi, 2000). While the “cycle” of evaluation was possible only to a limited extent in these current evaluations, this was the reason, for example, for the evolving pattern of interview questions as the evaluations proceeded.
One of the most useful features of realist evaluation is the categorisation of data collection methods into content, context, mechanisms and outcomes (Kazi &
Spurling, 2000: 8). For the current evaluations, these may be defined as follows.
• Content: the intervention or programme; here, it was the introduction of a mandatory Code, together with the requirements to become a signatory and later, the December 2002 legislation which changed the definition of an
“international student” (Education (Tertiary Reform) Amendment Act, 2002).
In the PD evaluation it was both the programme provided by Scott Strategic, and the needs analysis by Jan Hollway for Education New Zealand. For the purposes of both evaluations it is considered essential that both the time frame for the contracts, and finance available be seen as important aspects of the 'content'.
• Context: the situational factors which both pre-exist at the time of
intervention, and which develop during the intervention process. These are factors affecting the people and the institutions participating in the PD initiatives, and not those affecting the contractors and facilitators.
• Mechanisms: these are the motivations, practices (and sometimes goals), of the individuals and institutions who are the recipients of the intervention and/or the programme implementation. These mechanisms can be
enabling (of the intervention or programme), or disabling, and can arise out of reactions to the content or context, or from an interaction between both.
• Outcomes: the results of the intervention or programme. This is often seen as the major focus for funding agencies, but that is not the case when further funding also depends on how successful the particular processes have been in achieving these outcomes.
3.2 PROCEDURES
In what follows, the term 'visits' is used as a neutral term to include a wide range of activities: attendance at professional development (PD) seminars, interviews in educational institutions, a range of other interviews, and several further types of (face-to-face) data collection.
3.2.1 Activities / Visits
The original proposals accepted by the Ministry included 20 visits in each contract.
When the two contracts were 'combined' late in 2002, one of the arguments for this was that the combination would allow for more extensive data-gathering. A further argument was that the third round of PD, which took place in late 2002 and early 2003, could be a free add-on to the PD contract.
Both of these arguments were accepted, and both have been vindicated in practice. While a total of 40 visits were proposed (20 for each contract), by the end of June 2003 more than 80 visits constitute the key data base for the final reports. This was in part made possible following a decision not to survey
institutions by questionnaire, especially as the schools' PD had been surveyed by an evaluator working with Scott Strategic.
The sample has been a mixture of 'opportunistic', as in the unexpected chance to gather information at a recent two-day seminar in Nelson, and purposive, to
ensure a suitable range of both institutions, and meetings with key personnel. The Evaluation Team has also taken very seriously the following selection criteria for institutional visits:
• A wide geographical coverage;
• Coverage of all types/sectors involved;
• Coverage of institutions which are at very different levels of experience;
• The need to identify best practice in a wide range of areas.
This led to an initial purposive selection, initiated by the Team Leader with the help of the Code Administrator's office. A request was made for the Office to select two or three 'exemplary' Code applications for each sector. This was done on the basis that subsequent visits would be able to verify whether institutional practice was indeed that which was represented by the Code application. Despite
concerns expressed by some members of the Advisory Group about the reliability of such an approach, the Evaluation Team believed - and still believes - that it was a useful starting point.
After subsequently receiving 14 exemplary applications, it was found that these gave a fairly representative geographical sample, with Auckland figuring strongly, but with a number of other regions also appearing.
The procedure then followed was to contact one of the regional facilitators who had provided PD and to ask them for recommendations of highly regarded programmes in institutions other than the 'exemplary' sector. So, for example, a visit was to a location where the exemplary application was a secondary school, this might result in a primary school and a tertiary (public or private) institution also being visited.
This has clearly produced a biased sample, but this was a deliberate strategy.
The Evaluation Team saw little merit in visiting 'bad' institutions when the clear goal of the PD, the Code and the evaluations was to help to raise the standards.
On the other hand, it must be strongly emphasised that the findings of this part of the evaluation cannot be assumed to be 'typical' of institutions with IS. This was in part countered for the school sector, however, by a number of informal discussions with a wide range of experience and school background at the PD seminars
attended by members of the Evaluation Team.
The summary which follows indicates the nature of visits undertaken for the two linked evaluations.
3.2.2 Summary of Visits: Institutions
It was determined early in the evaluations to identify the industry as comprising seven main 'categories' of institution with IS programmes. These were primary and intermediate schools, secondary schools (including those with primary classes), polytechnics, universities, language schools (both private and state), private training establishments (PTEs) other than language schools (including both degree-granting and others, and state colleges of education.
Table 3.1 presents a summary of institutional visits for the two Evaluations.
Table 3.1 Visits to Institutions
Type No. of Visits North Is. South Is.
Primary schools 7 5 2
Secondary schools 10 4 6
Polytechnics 5 3 2 Universities 4 3 1
Language schools 6 5 1
PTEs 5 5 0
Colleges of Educn 1 1 0
Totals 38 26 12
As indicated earlier, a starting-point for these visits was the selection by the Code Administrator's office of 14 'exemplary' Code applications, comprising two or three for each sector. Twelve of these were visited. One refused on the grounds that they were "too busy", and in the other both the principal and the international director were new, and the school is currently preparing a completely revised Code application. They were happy to be seen later in the year if required.
The number of people interviewed, and the number of interviews varied from institution to institution. Typically, in primary schools the principal was interviewed, with on three occasions a second person present (in all three cases, the person responsible for ESOL). In secondary schools, the principal was normally
interviewed briefly, and the international director/manager interviewed separately.
In tertiary institutions, there was no clear pattern. In several cases two people were interviewed simultaneously. In four cases only one person was interviewed, but another completed a short questionnaire. In one language school three separate interviews were held, as well as a telephone discussion with the absent owner; in another, only the CEO was interviewed. Overall, 64 people were
interviewed and four senior staff sent in 'context' or other questionnaires. Thus, 68 staff provided data relating to the 38 institutions visited.7
The primary schools visited covered an interesting range of experience and numbers of full fee-paying international students (IS). They included two intermediate schools.
The secondary schools include two private schools, and represent schools with from fewer than 10 IS up to over 100.
The polytechnics include two major urban and three from secondary/rural areas.
One is a very significant player, with a long history of IS programmes The language schools comprise four private institutions and two University-affiliated schools.
The private training establishments include two degree-granting institutions, and three offering other forms of qualification. One currently has no full fee-paying IS, but is a signatory and planning to market for 2004.
The college of education has both IS students in 'standard' programmes, and visiting groups, often senior educators coming for short-term tailored programmes.
3.2.3 Summary of Visits: Professional Development
Table 3.2 presents an overview of professional development 'events' attended by one of the Evaluation Team in 2002 and early 2003.8
7 These figures do not include 'non-standard' interviews with five international staff
Table 3.2
Professional Development Events Attended
'Round' No. of Visits North Is. South Is.
Training 1 (3 days) 1 N/A
First Round 4 3 1
Second 4 2 2
Tertiary 1 0 1
Third 3 2 1
Total 13 8 5
The Training of the facilitators for the schools' PD took place over four days in Wellington, early in 2002. The Team Leader of the Evaluation Team attended three of these days.
The First Round was the series of one-day seminars for schools, covering a wide range of policy and other advice relating to IS programmes.
The Second Round comprised a range of regionally-determined follow-up
initiatives for schools, based on what the (regional) facilitators concluded from the First Round evaluations. In a large number of cases, there was considerable interest in the Code application process.
The Tertiary round was a limited series of eight PD seminars aimed at new providers in the tertiary sector. This programme was delivered by the TDO.
As noted earlier, the Third Round comprised seminars targeted at three distinct groups: primary schools showing as having IS, but who were not yet signatories;
primary schools who had not attended the earlier PD; and homestay coordinators based in or associated with secondary schools. In fact, this did not happen in any 'clean' way. A number of schools simply took this new opportunity to get further information and ideas about the provision of quality programmes for IS.
3.2.4 Other Interviews
Interviews were held with a number of people involved in the PD initiatives, although Code issues were regularly part of the conversation. The 'principal PD actors' - Jan Hollway (the TDO), Lyn Scott and Christine Scott (schools' PD contractors) - were all interviewed. Jan and Christine were interviewed four times each, and there was also a number of e-mail and telephone links with these two people. Lyn Scott was interviewed once. An interview was also held with the evaluator contracted by Scott Strategic to monitor their PD work in 2002, and e-mail contacts were also made with that evaluator. It should be added that all of the raw data collected for the schools' PD (seminar evaluations and
questionnaires), has been made available to the Evaluation Team, along with summary data of evaluations carried out by the TDO.
Interviews of various kinds were also held with nine of the fourteen regional facilitators for the schools' PD. This involved just five extra visits, as two of the interviews involved two regional facilitators participating at the same time, and two have already been counted in the list of institutional visits. Informal discussions with several other facilitators, however, took place in the course of the
observations of schools' PD sessions.
Other interviews which have been held are described below.
• Lester Taylor of Education New Zealand, which was contracted to the Ministry to investigate PD needs in the tertiary sector (Jan Hollway was then contracted to Education New Zealand to be the TDO and to carry out the needs analysis).
• A short-term contract staff member of the Ministry who was directly involved in the PD contracts.
• A brief and early interview with a contract staff member who assisted with the development and writing of the Code of practice, but was also involved
• Professor Colleen Ward, Victoria University of Wellington; a lengthy phone discussion in 2002 and a face-to-face meeting in 2003.
• Four face-to-face interviews, and e-mail/telephone contacts with the Code Administrator, along with phone conversations with Code support staff David Beer (who attended one face-to-face meeting), and Sarah Hart.
• Two meetings with Dr Nick Lewis, who is researching the development of three new industries in New Zealand (wine, education, fashion).
3.2.5 Other Visits
Five other data-gathering visits have been undertaken.
• Attendance at an ISANA regional seminar on Chinese IS, held in Auckland (August, 2002).
• Attendance at (and participation in) a seminar on Asians in Education, held in Auckland (October, 2002).
• Attendance by a contracted agent at the ATSA conference on the needs of IS, held in Nelson in May, 2003
• Informal and on-going discussions with a former9 Chinese doctoral IS, whose thesis is looking at intercultural communication in a period of increasing globalisation and a growing international knowledge economy.
• Participation in a project seeking to analyse IS student movements from Asia to New Zealand in the period 1945 to the present (project under the auspices of the New Zealand Asia Institute).