ProgressinEvaluationResearch: FSI97-203 discussionpaper '-080f£

'-080f£

discussion paper

FS I 97 - 203

Progress in Evaluation Research:

Documentation of Two Transfer-Workshops on the

"International Handbook of Labour Market Policy and Evaluation"

Günther Schmid, Peter Auer, Hugh Mosley, Klaus Schömann (Eds.)

April 1997

ISSN Nr. 1011-9523

The two Transfer-Workshops were funded by the European Commission, Directorate-General for Employment, Industrial Relations and Social Affairs (DG V)

We thank Miriam Hils and Karin Reinsch for invaluable editorial assistance.

~~~----~---- ---

.., ': ..."~ e 't".' '\

,'·i'; \;.~"~""" :.

¡~.!;."_~~ ~: '~1J

"~J;,

i;"i:~~tJ,

.(~tt...

.

^.'~ •

""il "~.

'~

ZITIERWEISE ICITATION

Günther Schmid, Peter Auer, Hugh Mosley, Klaus Schömann (Eds.)

Progress in Evaluation Research:

Documentation of Two Transfer-Workshops on the "International Handbook of Labour Market Policy and Evaluation"

Discussion Paper FS I 97-203

Wissenschaftszentrum Berlin für Sozialforschung 1997

Forschungsschwerpunkt:

Arbeitsmarkt und Beschäftigung

Research Area:

Labour Market and Employment

Abteilung:

Arbeitsmarktpolitik und Beschäftigung

Research Unit:

Labour Market Policy and Employment

Wissenschaftszentrum Berlin für Sozialforschung

Abstract

This report documents the results of two follow-up workshops on the International Handbook of Labour Market Policy and Evaluation. The aim of these workshops was the exchange of experience among academics and practitioners in the evaluation of labour market policies. The Handbook project as well as the two workshops were supported from the beginning by DG V of the European Commission through the Employment Observatory and particularly the MISEP network. The first workshop was held in May 1996 at the Social Science Research Center in Berlin and concentrated on the topics of methodology and monitoring. The second workshop took place in November 1996 in Lisbon and focused on three policy questions: long-term unemployment, lack or mismatches of skilled human resources, and inflexible working time arrangements. The workshops were attended by two or three representatives from all Member States of the European Union. The workshops and their documentation are organised in three parts: Introduction into the subject on the basis of selected chapters of the handbook; presentation of best practice cases; discussion and exchange of experiences. The report is complemented by the list and adresses of participants and the contents of the

handbook. . .

Zusammenfassung

Dieser Bericht dokumentiert die Ergebnisse zweier Workshops über das

'International Handbook of Labour Market Policy and Evaluation'. Das Ziel

dieser Workshops war der Austausch von Erfahrungen zwischen

Wissenschaftlern und Praktikern in der Evaluierung der Arbeitsmarktpolitik. Das

Handbuch-Projekt sowie die zwei nachfolgenden Workshops wurden von

Anfang an von der DG V der Europäischen Kommission im Rahmen des

Beschäftigungs-Observatoriums und des MISEP-Netzwerks unterstützt. Der

erste Workshop fand im Mai 1996 am Wissenschaftszentrum Berlin statt und

konzentrierte sich auf die Fragen der Methodologie und des Monitoring. Der

zweite Workshop wurde im November 1996 in Lissabon abgehalten und

beschäftigte sich mit drei Problemen der Arbeitsmarktpolitik: Langzeit-

Arbeitslosigkeit, Mangel an qualifizierten Arbeitskräften und Ungleichgewichte

zwischen angebotenen und nachgefragten Qualifikationen, inflexible

Arbeitszeitarrangements. Teilnehmer an den Workshops waren jeweils zwei bis

drei Vertreter aus allen Mitgliedsländern der Europäischen Union. Die

Workshops und deren Dokumentation sind in drei Teile gegliedert: Einführung

in das Thema auf der Basis von ausgewählten Kapiteln aus dem Handbuch,

Präsentation von 'best practice'-Fällen, anschließende Diskussion und

Austausch von Erfahrungen. Die Dokumentation wird ergänzt durch eine Liste

der Teilnehmer nebst Adressen und Inhaltsverzeichnis des Handbuchs.

International Handbook of labour Market Policy and Evaluation: A Follow-up

Günther Schmid

1

Table of Contents

Workshop I: Monitoring and Methodology, Berlin, May 13-14, 1996

Monitoring of labour Market Policies Introduction

Peter Auer , , , , , , , ,..,.., ,4

Presentation of Best Practice: The Case of Sweden

Staffan Nilsson , , " , " 1 O

Summary of the Discussion

Peter Auer .., , " , "" ".,., " 12

longitudinal Designs in Evaluation Studies Introduction

Klaus Schömann., , "" " "" ,.", ,.,..,.18 Presentation of Best Practice

1. The Case of Luxembourg: Uwe Warner 22

2. The Case of Germany: Jürgen Kühl 24

3. The Case of Spain: Felipe Saez , , 28

4, Other International Cases: Robert Fay, 30

Summary of the Discussion

Klaus Scnômenn. " , , , , _, , , 33

Experimental Evaluation of European labour Market Policy Introduction

Anders Björklund ,.., , ,.., ,.,., ".", ,,39

Presentation of Best Practice:

The Case of United Kingdom: Michael White , 43

Documentation of the Discussion

Miriam Hils, Günther Schmid 47

labour Market Policies and long-term Unemployment

Workshop II: Labour Market Policy,

lisbon, November 8-9,1996

The Significance of PES Market Share for Target Group Policy

Jaap de Koning

71

Summary of the Discussion

Hugh Mos/ey 82

Wage Subsidies and long-term Unemployment

Employment Subsidies for the Lonq-term Unemployed

Bernard Gazier 86

New Job Market Policies for the Particularly Disadvantaged Unemployed

A/fans Schmid 91

Evaluating Wage Subsidies for the Long-term Unemployed in the UK

Nigel Meager

97

Summary of the Discussion

Peter Auer

106

labour Market Training Against Unemployment

Evaluation of Labour Market Training

Kleus Sch6mann

111

What Works, Who Works? The Impact of Active Labour Market Programmes on the Employment Prospects of Young People in Ireland

Philip J. O'Connel! and Fran McGinnity 115

Evaluation of Training: The Spanish Experience

Felipe Saez

120

Summary of the Discussion

Günther Schmid

126

Flexibilisation of the labour Market

Evaluating Short-time Work Schemes

Hugh Mosley 136

Evaluating Short-time Working in Italy

Leonello Tronti

142

Evaluating Worksharing in Canada

Ging Wong : 154·

Summary of the Discussion

Hugh Mas/ey

162

Towards a Community Databank on labour Market Policies: Work in Progress

Laura Bardane

165

Summary and Conclusions of the two Workshops

Progress in Evaluation Research

Günther Schmid 174

Conference Hand-out of Participants:

National Systems of Monitoring Labour Market Policy

Monitoring and Evaluation Research in the Employment Services in

Germany

Christian Brinkmann 186

Monitoring and Evaluation of Labour Market Policies in

Germany

JOrgen Warnken 190

Monitoring of European Social Fund (ESF) Projects in Labour Administration of

Finland

Torsten Kolehmainen 193

Monitoring of Labour Market Policy and its Implementation in

Finland

Heikki Räisänen 196

Monitoring in the

Austrian

Public Employment Service

Marius Wilk ; 200

Employment Service Systems for Follow-up of Employment Policies (Monitoring and Evaluation) in

Great Britain

Andrew Birfwhistle 202

Monitoring and Evaluation of the Labour Policies: The

Italian

Situation

Luciano Forlani 206

The

Irish

System of Monitoring and Evaluation of Employment Policy

Roger Fox 208

Monitoring of Labour Market Policy in

Sweden

Bengt Härshammar 210

The Monitoring and Evaluation of Employment Policies in

Spain

Felipe Saez 213

Follow-up System on Employment Policies in the

Netherlands

Leo Schrijver 216

Annex 1:

Adresses of Participants of the workshops 219

Annex 2:

Contents of the International Handbook of

Labour Market Policy and Evaluation 225

Introduction

International Handbook of Labour Market Policy and Evaluation: A Follow-up

Günther Schmid

In the following we present the results of two workshops on the International Handbook of Labour Market Policy and Evaluation.' The aim of these workshops was the exchange of experience among academics and practitioners in evaluation of labour market policies on the basis of selected chapters of the Handbook. The Handbook project, as well as the follow-up workshops, were sponsored from the beginning by DG V through the Employment Observatory and particularly the MISEP network.' The first workshop was held in May at the Social Science Research Center in Berlin and concentrated on the topics of methodology and monitoring. The second workshop took place in November in Lisbon and focused on policy questions and on the three main problems which have plagued most European countries for almost two decades: long-term unemployment, lack or mismatches of skilled human resources, and inflexible working time arranqernents."

Whereas the aim of the Handbook was to take stock of the state of the art in theories and methods of evaluation research, the objective of the two follow- up workshops was to complement this rather academic undertaking by

1.

presenting best practice cases from various countries of the European Union (EU),

2. exchanging experiences and views between practitioners and consumers of evaluation research, and

3. by developing a kind of European common consciousness of professional standards and awareness of institutional requirements for the evaluation of labour market policy.

We invited representatives from all European member states so that participants from countries which have had very different experiences with evaluation research can share these in an open forum. Some countries have a long tradition of evaluation, while others are just beginning to conduct such

2

Schmid, Günther and Jacqueline O'Reilly, Kiaus Schömann (eds) 1996, international Handbook of Labour Market Policy and Evaluation, Cheltenham: Edward Elgar, 954 pp.

We want to express, on this occasion, our gratitude to Armindo Silva and Sergio Piccolo for their enthusiastic promotion of this project.

We are deeply indebted to Karin Reinsch for her splendid help to organize these workshops and her inimitabie humour.

3

research; some countries have a more academic tradition of evaluation, while others have a more institutional tradition. Apart from the experts who presented best practice cases, we invited the workshop participants on the basis of the following criteria:

1. To ensure that the agencies that develop policies and representatives from agencies which implement policies were represented, one delegate from the public employment service and one from the labour ministry were asked to attend. This was not, however, possible in all cases because the responsibilities are not clearly separated in all countries.

2. To provide the necessary feedback on the practical implementation of the results of evaluation research, we invited people who are responsible for managing evaluation research and, at the same time, are participating in the policy formation process. Such people belong really to a very rare species, and we were happy to have found them."

4 See the list of participants at the end of the Handbook Follow Up.

This Handbook follow-up is structured in the following way:

1. summaries of chapters in the Handbook are presented as a means of introducing the subject of discussion,

2. best practice cases are presented, followed by the comments of panellists, 3. a general discussion reflecting and sumrnarisinq the main topics and results

follows, and

4. some conclusions from the two workshops are drawn, along with suggestions for further research and procedures in institutionalising good monitoring and evaluation.

In the wake of this project, the future of labour market policy and evaluation can no longer be considered solely as a national undertaking. Increasingly, it is possible to think in terms of a European evaluation process with an European content. Due to the ongoing globalisation of the economy and its impact on world labour markets, the use of trans-national contents in the evaluation of labour market policy is becoming more important.

Thus, a second generation of European labour market policy and evaluation has arrived, and we hope that the International Handbook of Labour Market Policy and Evaluation and this follow-up report will support and enhance this process.

2

Workshop I:

Monitoring and Methodology

in Berlin

at the Wissenschaftszentrum Berlin für Sozialforschung (WZB)

May 13-14, 1996

Monitoring of Labour Market Policies

Peter Auer

Introduction

In recent years the monitoring of labour market policy has become a major issue for programme implementation, both at the individual EU member country level and the European level. The reasons for this include:

- Fiscal constraints, lean administration and the reform of the public sector.

New ideas about public service involving a shift from bureaucratic and centralised rule-making toward decentralised and market-oriented responsibilities and assessment systems, as well as a general awareness of public spending (accompanied by increased auditing by controllers), have contributed to the pressure to set up monitoring systems. Together with decentralisation (see below), the introduction of private-sector management methods (e.g., management by objectives, "profit centres" or "cost centres") in the public sector has also compounded the need for assessment of results. Partially related to that need is an increasing number of laws and programmes which have a "built-in" obligation to monitor and evaluate the performance of labour market programmes.

- The decentralisation of LMP implementation. Handing down responsibilities for LMP programmes to lower administrative levels, and thus reducing the control opportunities inherent in centrally administered programmes, could complicate LMP monitoring. But decentralisation promotes the introduction of monitoring systems in two ways. First, it provides local agents with the information they need. Second, it provides central bodies with information on their local branches, enhancing their control scope as they can compare the results of their local agents. It also provides better targets for local agents by providinq them with results obtained by other agents. The trend toward expanding the discretionary power of decentralised levels (e.g., the latitude that local employment offices have for deciding which measure, such as job creation or job training - is suitable for their clients) also entails increased monitoring. After all, local agents must justify their allocation choices.

- Increased ESF financing for national LMP and provisions for fol/ow-ups. The reform of the structural funds in 1989 led to multi-annual programming and greater involvement of the Member States in monitoring the implementation and effects of measures co-financed by the ESF. Because, as ESF funding is integrated in the financing of existing measures, ESF co-financing often does not lead to the design of new measures. Thus, ESF (co)financed 4

1

measures are to a considerable degree identical with national labour market policy measures and - in principle - there should be no difference in ESF and national LMP monitoring. However, there is considerable cross-country variation here. ESF-financed measures are not a specific subject of this article, but, because of the overlap, both forms of monitoring are often linked (for ESF monitoring see the work of the Centre for European Education Expertise in Lyon and the MEANS Programme of the European Commission).

- The European Commission's efforts to make economic growth more employment intensive - above all through the activation of labour market policy (European Commission, 1994) - make monitoring more necessary than ever. In the wake of numerous European Council meetings, and especially since the resolutions of the European Council meeting in Essen (December 1994), employment and employment policies must be monitored by member countries and the Commission, which has to report on developments every year from late 1995 onwards. The Commission, with the help of the member states, Eurostat, and its employment observatories (MISEP, SYSDEM), intends to establish employment policy indicators to permit follow-up work on national policies in financial and physical terms.

These activities, if successful, might also lead to an increase in national monitoring.

- Finally, technological progress has made monitoring possible at all levels of organisations, and data transfer through data networks has increased tremendously the potential scope and speed of monitoring. These changes have paved the way for setting up more sophisticated monitoring systems.

The questionnaires and selected expert interviews create the impression that all EU member states are currently engaged in setting up or expanding monitoring activities. Labour administrations are trying to follow the implementation of LMP

more closely than used to be the case, but it not possible to clearly place the

^I

different countries on a scale ranging from "no system" to "fully implemented

I

system". The one exception is Sweden, which we would place at the top of the list, although even there the system is not yet fully in place. Otherwise the condition of monitoring is, as yet, rather heterogeneous even within countries;

sophisticated monitoring of one measure might contrast with the absence of

follow-up procedures for another. This "state of the art" of monitoring was then

discussed using an overview on monitoring practices in the EU member

countries. (Table 1)

OrganisationContinuousinformationon*:Monitoring CountryMonitoringCoordí-Decentra-localIncent-SanctionsBudgetPartlei-Post-participationTratningEmploy-LevelofTrendObservations InstitutionsnationJísationBudget'{vespatíonprogrammentdevelop- among(Decon-discretion

-

pro-ment lnstítu-centra-motion- tiansfion) employ-qualffica-earnings menttion BelgiumMol,ONEM,VDAB,+++++--n.a.n.a,n.a.n.a.n.a.n.a,n.a,+-7RegionalparticularismhindersoveraU FOREM,ORBEMmonitoring,monitoringactivities regional. Denmarkn.a. GermanyBA,IAB+++++++--++++++--+++++ 71...Strongsystemdevelopment;notail measuresmonitored. GreeceMol,OAED,PIEKA+++--+++---++++ 71+Developmentofregionalemployment observatory, SpainMol,INEM,++++--++++--++++ 71+FORCEM FinlandMol+++++++--++++++++--+++++++ 71++Highstageofdevelopment(MbR) FranceMol,ANPE+++++++-+++++--++++++ 71++Notallmeasuresmonitored;both ANPEandMolareactivelydeveloping thesystem IrelandDEE,DSW,FAS+++++--++++++++++++++ 71+ Italyn.a. luxemburgn.a, NetherlandsLBA,CSA,RElA+++++++--++++++-++++++ 71+ AustriaMol,AMS+++++--T++++--++++++ 71++NewAMSisdevelopingmonitoring. PortugalMol,IEFP+++--++++++T-++++ 71+Developmentofregionalemployment observatory. SwedenMoL,AMS++++++++++-+++++++++-++++++ 71+++Highstpgeofdevelopment(MbR) UnitedDEE,ES,lC+++++++(+)+T++++++++++++++ 71+Kingdom --- *atleasteverysixmonths+=low++=medium+++=high -negligiblen.a.=informationnotavailable ++-=thoroughlymonitored +=lessthoroughlymonitored**fortrainingprogrammeoremployment promotion Source:Questionnaire,selectedexpertInterviewsandrelateddocuments,ownclassification;UK:providedbyMISEPcorrespondentdirectly. --"--~"~~""~._"-~-"~~-~-~-o

It appears that there is a north/south divide in monitoring, and that many (especially southern) EU countries clearly experience problems in setting up monitoring systems. Although certain problems are common to all countries, some also face complications relating to their particular administrative structure and other specific factors. Besides problems of co-ordination between different agencies in charge of monitoring, it is the absence of clear goals in employment policy measures that often makes monitoring difficult. Here, the often contradictory relationship between short-term quantitative goals (the prime interest of politicians especially prior to elections) and long-term qualitative goals (whose fulfilment is often targeted by those directly in charge of the participants) plays a role. Thus, for example, centrally established quantitative volume goals create problems even in the traditionally more bottom-up decision-making channels of Swedish labour market policy. A more technical problem that was also reported was the lack of adequate data (or difficulties in gaining access to such data) from which to construct indicators. Also, the lack of personnel and financial resources was seen as a problem, as was the acceptance of monitoring by those whose work was monitored because of the element of control involved. Thus, it is important to show that monitoring is not an instrument of control, but a way of enhancing the efficient working of the organisation.

Finally an "ideal-typical" monitoring system was discussed. Such a system should consist of:

- input, output and outcome goals specified by the political and administrative authorities at the national, regional and local levels in a bottom-up procedure;

- a definition of indicators that allow goals to be measured;

- the actual monitoring process based on 1 and 2 and on statistical information on financial and physical indicators relating to 1 and 2; and

- feedback loops to ensure that observed irregularities are addressed adequately.

Figure 1 illustrates' such a monitoring model. It starts with policy formation in which programmes and individual measures are chosen. Even at this early stage, clear goals for the programmes must be defined, and indicators enabling progress toward those goals to be measured must be established if monitoring is to be effective. In other words, effective monitoring must be written into laws and ordinances (as is the case in the French five-year employment law of 1994). After programmes have been implemented, a continuous process of observation should begin, the intervals of which may vary from measure to measure. A quarterly observation period seems a reasonable compromise, but for short-term programmes, shorter periods might be appropriate. The core function of monitoring is to detect indicators turning "red" and to initiate subsequent remedial action (feedback). Feedback between monitoring and

Figure

1

Monitoring of labour market policy

I Policy formation

I~

r--

^I ~ ^{choices ~.}

I

I

I

_{I :}

I

'; Programmes measures

I

_o¿

i ~ ~ Goals ~ ^{.' .}

I

^"Tl

o

! ~

Indicators

i

cu

I

^(D

.o _j ^(D

u

I

I Q.

G) I 0-

G)

!

^œo

u, 7\

I i ^I

..~ Programme

~

I

implementation

I

I

Labour market policy

1

I

MONITORING

" :~Financial monitoring 1

,.1--.L...---''---,L<I(--i~

Performance monitoring2

¡

I

Evaluation

¡

~.---~~~

I

Labour market

monitoring ^¡--

Notes

1 Controlling spending and the number of clients served in relation to the goals set.

2 Observation of selected result indicators (e.g. post-participation employment).

l

\

j I

programme implementation is stronger than between monitoring and policy formation because of the role of monitoring as an instrument linked to programme implementation. However, the results of monitoring in which the standard performance of programmes is measured should also be fed back to policy formation. Evaluation, in which deadweight, substitution effects and other factors are also taken into consideration, offers a more profound assessment of programme impacts than monitoring, and thus affects policy formation more strongly than monitoring.

However, monitoring should leave only those indicators which cannot be observed directly and regularly to evaluation. A relatively simple monitoring system will allow for a national/regional/local breakdown in order to facilitate comparisons between agencies. For any given programme at any given administrative level, such a system should at least be based on the regular (monthly, quarterly) observation of:

- the budget allocated and the resources spent (breakdown by spending categories, target groups and target sectors if appropriate);

- planned participation and actual participation (breakdown by target group and target sectors if appropriate);

- costs per head (per hour) on the basis of 1 and 2 and breakdown by spending categories, target group/sectors if appropriate.

Thus, monitoring systems would provide useful information on financial and physical take-up and costs, and under-performers could be detected by comparing regional/local units. At the same time, the differences in labour market conditions in local areas should be taken into account and cost factors should not be overstated, as they might lead to creaming. A link between labour market monitoring and labour market policy monitoring is also part of the model:

results of LMP have an impact on the labour market and an ideal monitoring

model must establish this link.

The Case of Sweden

Presentation of Best Practice

Staffan Nilsson from the county labour market board in Kristiansstad presented his monitoring system installed on a laptop computer as' a best practice case.

The system was developed by the county labour market board and has two parts: one called Vaksam (control), the other Verksam (performance), the first being for financial planning and monitoring, the second for results monitoring.

The system was developed after the entire organisation (AMS) was decentralised in the early 90s, as it appeared that there was no nation-wide monitoring system installed to support the new decentralised delivery of services. The system was created for the heads of the local employment services, and it shows all local offices of the region but has no nation-wide scope. It is only possible to plan on the level of the local unit, but for the purposes of monitoring users have access to all local offices of the county. It is also possible to monitor the plan and the outcome for the whole county aggregate. The system will be replaced by a nation-wide system called PRESTO, for which Vaksam was a model.

With the help of the system, it is possible to compare on-line plans and outcomes. Since the system has been in use the matching between plans and outcomes has been improved. Planners are now within 2% of the original budget. The planner's job is, however, not only to steer the budget, but also to reduce unemployment. This demands a high level of strategic knowledge and analytical skills from the local heads. So Vaksam is merely a tool, the successful use of which depends overwhelmingly on the knowledge and skills of local heads.

An example for the use of the system: a labour market training measure has an estimated 110 participants in October, and the total budget for the measure is 97 Million SEK. To change the number of participants, the new number is typed into the spreadsheet, which then calculates all the numbers which have to change automatically. Basically, the system compares financial outcomes with plans and indicates to planners whether targets can or cannot be reached.

In terms of performance monitoring (Verksam), it is also possible to compare plans with outcomes. For example, one could look at how large a share of the unemployed found a job last month through whatever means, and compare that result with the goal set in the activity plan for the local office. The 10

Staffan Nilsson concluded his presentation by stressing the fact that the monitoring system can be considered the hardware of the planning process, while the software is the organisation. The most important factor in the implementation of the system is the creation a new culture in the organisation which allows for the effective decentralisation of resources and responsibilities.

90% of the success of the system is related to the culture, and only 10% had to do with the computer system.

goal set was that 7% of the unemployed should get a job each month. The goal was reached in September only. This demands further analysis and discussions: Was the goal badly formulated, or did something in the market changed so dramatically that it was impossible to keep up the levels?

Monitoring of performance is divided into three categories: unemployed,

vacancies and special target groups. There are specific graphs for each group.

Peter Auer

Summary of the Discussion: Monitoring Labour Market Polley"

Ma.rius

Wilk

from the Austrian Employment Service underlined some of the difficulties which the AMS (Arbeitsmarktservice) has with monitoring. A general question arose: Is it possible, given the degree of decentralisation which now prevails in Austria, to steer the local levels by means of pre-determined goals, such as specifying numbers of people to be placed in measures, or a post- employment target? The problem is that the effectiveness of programmes is usually not known in advance, and that a given labour market situation might change rapidly. It is therefore difficult to set labour market goals in advance.

However, the AMS has started a process of discussion with its regional managers. After the first three months of this monitoring experience, it will be possible to determine if the goals have been reached. It is necessary to discuss whether the measures which currently exist represent an efficient means of reaching the goals determined in advance. The AMS has a monitoring system which allows for a quarterly review of certain indicators (a monthly review is also possible, but this period is too short for performance measurement) but it is, as yet, less sophisticated than the system in Sweden.

Margarida Abecasis from the General Directorate for Employment in Portugal noted that while the Portuguese system is currently very centralised, decentralisation is now on the agenda. Portugal has also developed monitoring for measures co-financed by the European Social Fund, and the setting up of this system was aided by the latter. She also explained that an "employment and training observer" which monitors local labour market developments had recently been created. In addition, information on vacancies and placements gathered at all Portuguese job centres is been linked, and there is currently an effort underway to develop this system in order to use it for the monitoring of employment and training measures as well.

Heikki Räisänen from the Finnish ministry of labour (see also paper in appendix) explained that the Finnish system of monitoring is decentralised and centralised at the same time, linking the information of all 184 job centres. Both stock and flow data on the number of participants in programmes and set goals are monitored, although this is somewhat problematic given that the goals, as well as the budget, are set by the parliament. The Ministry of Labour negotiates regional targets with the 13 regional offices, which then negotiate their goals with their local offices. Not only the goals, but also the also cost plans are monitored, although this is difficult because unit costs depend also on the (training provider) market conditions, and these costs vary by region. As costs cannot, therefore, be very closely monitored, it has happened that not all the

5 See Appendix for the description of National Monitoring Systems in the EU.

¡ ¡

¡

"

I

12

Staffan Nilsson

pointed out that in Sweden, as well, local labour offices are required to stick to goals pre-determined in negotiations. Only when there are dramatic changes can goals be changed. On the regional level, as well, goals are set in negotiations with the central level. A change is now underway in Sweden, however, because the parliament has set volume targets. This implies a certain recentralisation of the system, a tendency which he currently observes in many countries.

resources had been spent at the end of the year. This is embarrassing given the high level of unemployment in Finland.

For the PES, the share of filled vacancies and the time needed to fill them are important indicators. The market share is also observed, but vacancies filled by internal recruiting in firms are not reported. An important indicator for training is the target group share and student working days (8 hours). Then the employment rate is monitored after completion for all measures (2, 3 and 6 months after completion). Negotiated goals can only be changed every 6 months (during the semi-annual negotiations).

Didier Gélot

from the French Ministry of Employment and Social Affairs noted that despite the strong French tradition of centralised state intervention, the follow-up system on employment policies in France is relatively decentralised, involving many actors. The introduction of a more centralised system had been considered, but this development was blocked due to budgetary reasons. On the other hand, training has been decentrallsed, and the ministry has lost some information it formerly had. The monitoring of training can be considered to be in a phase of transition. Another difficulty is that no data for units smaller than the

départements

are collected, although it is important that follow-ups occur shortly after the implementation of policies, and that policies are designed in accordance with local needs.

However, the ministry establishes so-called

"tableaux de borä',

which permit a physical and a monetary follow-up. However, participant and financial follow-ups are dealt with separately, although their joint consideration would, of course, be ideal from the point of view of monitoring. We are also in a phase of simplification of statistics. Mr. Gélat expressed interest in the goal negotiation process in Swedish LMP, and wanted to know if there are conflicts in this process.

Staffan Nilsson

could not recall a situation in which there was a large gap between the goals of the regional and the local levels. The object of the negotiations is to even out these gaps. If the local situation is very atypical, this fact will be taken into consideration during the resource allocation process.

Luciano Forlani

explained that the Italian situation is somewhat different.

In Italy, there is strict centralisation (except in the area of training, which is the

responsibility of the regions; and where monitoring exists, especially when ESF financing is involved) of employment policies. There is an "employment observatory", which is a centralised structure with links to regional and local bodies of the ministry. However, this will change with the arrival of the new government, which is in favour of administrative and institutional decentralisation. There will be a strong role for the regions. As far as monitoring and evaluation is concerned, there is the multi-annual programme to monitor the five action areas of Essen. In the field of industrial relations(where monitoring of wage-indexing is important) as well, a decision was reached among the social partners to launch an evaluation of employment policies. The instruments and methods to carry out an annual evaluation therefore exist. The experience of ESF evaluation and monitoring practice on the territorial level is another source of input. As yet, however, Italy only has a provisional evaluation system, which employs simple methods. It will be important to determine relevant indicators and standards. There is also the question of the choice of the appropriate institutions for evaluation and monitoring. The issue of how to implement of monitoring and evaluation is a political as well as a technical question, and it should be considered in all its complexity.

Sergio Piccolo of the European Commission also pointed out the fact that monitoring/evaluation has a political side. He also observed that there are many programmes which have a more indirect impact on employment. Is there also an ex-ante evaluation of the employment impact of such programmes, as in the case of the programme for the suburbs in France?

Didier Gélot replied that there are few ex-ante evaluations in general, nor does one exist for the programme in question. However, a commission (mission d'analyse économique) has been set up with the principal objective of facilitating dialogue with the Ministry of Finance. This commission is currently the only body capable of running models for programme impact forecasts.

Peter Auer from the Social Science Center noted that in his research, the problem of overspending/underspending due to the lack of monitoring had been mentioned as a problem. He asked if the panellists also saw this as a problem.

All the panellists agreed that the question is important, and that, on the regional level, a monitoring system such as the Swedish' one helps to prevent the failure to meet budgetary targets. In Finland, underspending is frequent and monitoring often comes too late to permit resource reallocation. Similar problems were reported from Portugal and Austria. Robert Fay from the OECD said that overspending/underspending is not necessarily a bad thing, but that it may be when it comes to reallocating funds. Also, if creaming exists, then meeting the targets may become more important than helping the unemployed.

Klaus Käding of the European Commission also discussed the problem of creaming. If there are systems for the management of results but no clear

14

target group objectives, this will lead to the preference for those less hard to place candidates. It is much easier to attain goals by serving the less needy.

How do countries handle this?

Marius Wilk

noted that Austria has special aims for target groups, but he added that it is possible to have common policies for target groups and non- target groups.

Heikki Räisänen

added that in Finland, certain target groups receive special attention: there are national targets which depend on local circumstances.

Staffan Nilsson

added that the budget alone should not be the goal, but rather achieving the best results for the target groups

Kay Hallahan

from Ireland asked if there had been feedback from the monitoring process of goal-setting at the national level, and if qualititative goals also exist.

In his response,

Staffan Nilsson

pointed to the fact that his county has had the lowest unemployment rate in Sweden over the last two years, and that this was at least partially due to the monitoring system. It proves that the system has an impact.

Armindo Silva

from the European Commission underlined the difference between monitoring of LMP in the member states and monitoring at the European level. First, at the European level, there are no pre-defined volume goals. In addition, the feedback-loop, important for national monitoring, does not exist on a European-wide level for obvious re,asons. Employment and labour market policies are basically national policies. At the European level, there is also a broader macro approach, and not a micro approach. However, as far as monitoring is concerned, the Commission has the more qualitative reporting systems such as MISEP, and indicators will now be developed together with EUROSTAT as a quantitative instrument. For that purpose, a task force was set up by EUROSTAT consisting of representatives and experts from 6 or 7 member states and the OECD, which will produce a feasibility report by the end of the year. Mr. Silva asked if the establishment of such a data base was unrealistic, or is it indeed feasible?

Christian Brinkmann

from the Institut für Arbeitsmarkt-und Berufsforschung answered that, in order to be meaningful, the indicators have to be very carefully selected. Such indicators can easily be biased, and should refer to qualtitative aspects. In Germany, there is also a tendency to prefer short-term direct labour market integration indicators as a sole effectiveness measure, which is very one-sided.

Helmut Höpflinger

brought up the issue of creaming again, and

Heikki Räisänen

thought that, while there will always be creaming, careful targeting

will help to alleviate it at least in part.

Didier Gélot

added that in France, there

have not been many studies on creaming, but that there is a spontaneous

development in the labour markets which has the effect that, increasingly, the better-qualified find entry into schemes, especially to youth schemes. Julius op de Beke from the European Commission noted that the easy-to-place have good gross employment effects but bad net effects, while it is the reverse for the hard-ta-place. If monitoring systems can deliver information on that question with a sufficient degree of detail, then something can be done about it.

Peter Auer told the participants that in the largest employment and training programme in the US, the Job Training Partnership Act (JTPA), cost effectiveness is no longer a monitoring indicator because it had led to creaming in the past. Post-employment indicators currently prevail. Günther Schmid of the Science Center said he thought that only a closer link between budget allocation and performance outcome could solve the question of creaming. If longer-term indicators such as stability of employment in terms of job tenure and wages are not linked to more immediate monitoring indicators, then the latter will always support a trend towards creaming. Schmid asked to what extent performance indicators play a role in the Swedish budget allocation process. He also asked if there is an incentive for local and regional managers of the labour market administration to increase activity rates. The latter could be a useful monitoring indicator. In terms of the underspending/overspending question, Schmid sees the danger that, if underspending continues, it could lead to a decline in the budget because leftovers would be interpreted as money not needed. Is there an interregional spending compensation mechanism (some overspend, some underspend)?

Staffan Nilsson responded that performance indicators playa minor role in budget allocation. There has been a large discussion in Sweden on that issue, but complex indicators have to be developed to come to grips with the problem.

Jürgen Warnken from the German Ministry of Labour added that there have to be mixed indicators measuring short-term integration into the labour market, as well as other longer-term and more qualitative elements. Roger Fox from Ireland (FAS) saw a problem in the fact that there are so many different objectives behind these programmes. If only the disadvantaged were helped, then it would certainly be possible to find indicators. At the same time, however, the goal is also to lower unemployment and to eventually stimulate economic growth through training and other measures. In general, said Mr. Fox, it is perhaps better to have no monitoring system at all than to have a very simple short-term one. Thus, the trend should be to move towards more complicated and more extensive monitoring. Building up qualitative indicators is important but, at the same time, very difficult.

Uwe Warner from Luxembourg noted that Luxembourg is a very small country, but has a labour market bigger than its population of working age, because of the "frontaliers" who live in neighbouring countries but work in Luxembourg. This means that it is necessary in Luxembourg to look at developments in Belgium, Germany and France. A system of indicators

16

II

_,!

I': )

I

^j

allowing for cross-country comparisons, e.g., the impact of other countries' measures on Luxembourg, would be appropriate.

Longitudinal Designs in Evaluation Studies

Klaus Schömann

Introduction

Evaluations of labour market policies cannot stop after the first step of monitoring of participants and controlling of financial flows of subsidies to employers or employees. To be informative about the impact of labour market policies on firm or employee behaviour on the labour market, evaluations of labour market policies need to be more theory-driven. This basic understanding of evaluation has implications for the methodology applied in evaluations and the design of evaluation studies. Most labour market theories or other theories with relevance to processes on the labour market state hypotheses concerning desirable or undesirable changes in labour market outcomes. In their simplest form, these theories make predictions about the expected result of participation in a certain labour market programme. Such a hypothesis may state, for example, a likely reemployment probability after programme participation. This session is, therefore, dealing with questions of how such hypotheses can best be tested and the implications for data collection.

Defining the Longitudinal in Evaluation Studies

Frequently evaluators are called to carry out evaluations ex-post. These evaluations start after the expiration of a programme. In such cases they usually have to rely on existing data sources and surveys to measure programme effects or apply a retrospective data collection. This session, however, will focus more on ideal-typical designs of evaluation studies in order to

raise awareness for critical design features

of

evaluations.

More emphasis must be put on the design of evaluation studies to allow for adequate comparisons of at least two points in time: before and after programme participation.

If programme administrators select those whom they consider to be the best-qualified among the unemployed for a high-quality programme, estimates of effects of this programme are - without controlling for such positive selection into the programme - likely to overstate the effects of this labour market measure compared with a situation in which no such selection of participants is applied. In the methodological literature on evaluation this problem is called selection bias. Results on the efficiency of a measure are only a valuable basis

18

,¡

,r

I

¡

~I'.

of information for judgements on programme effects if these prior conditions are respected and dealt with in estimates of programme effects.

A related methodological problem in obtaining uncontaminated programme effects deals with the issue of uncontrolled heterogeneity in a sample which is the basis for estimating programme effects. Unobserved heterogeneity which is not controlled for in the estimates could indicate the presence of a state dependency. This is the case, for example, if the probability of recurrent unemployment appears to be high once unemployment has been experienced for a first time. However, previous unemployment might only be a proxy for other unobserved characteristics which are correlated over time. If such underlying processes remain undiscovered, evaluation studies are very likely to report misleading results and any policy conclusions derived from these results have an unclear empirical basis.

Based on the theory-driven aspect of longitudinal evaluations, our definition of a longitudinal design originates more in advances in longitudinal methods and analyses of causation applied to longitudinal data than in the survey technique perspective. Longitudinal studies are defined as theoretically grounded analytical studies using data which contain explicit references to the time structure of information. The exact recording and analysis of the time structure of information is, in fact, a crucial element of evaluations.

Panel Studies

The broad definition of the longitudinal element in evaluation studies includes studies which make at least two observations of the same participants measured at different points in time. In some instances, the time lag between the first and second contact may be very short. Examples of such studies are follow-up studies which are interested in changes after a specific intervention or the occurrence of a certain transition (e.g., school to work or exit from the labour force). Typically such studies would interview participants in labour market programmes once before entry into the programme and a second time some months or years after completion of the programme. On the other hand, such "minimal" two-wave panel studies can involve a considerable length of time.

Table 1 The Structure of Panel Data

Points of Dependent Independent Variable Programme Observation Variable (Time Dependent or Participation"

Independent 1 ^"

t

1 Y1 X1

0

1

t

2 Y² X²

O

2

t

n Yt Xt Dt

1. An example of a time dependent variable may be labour earnings which change between different points of observation; a time independent variable is the date of birth which rests unchanged during the whole period of observation.

2. Programme participation occurs is recorded at discrete points in time usually the variable named D takes values of zero or one at each point in time thus denoting programme participants (see also chapter two and three).

A change of perspective occurred in the 90s, when many researchers perceived that not only panels focusing on the supply side of the labour market are needed, but also longitudinal assessment of the demand side; i.e., the firms as potential users of labour market policies (Projektgruppe Betriebspanel,

1994). Beyond these numerous firm level panels for evaluating economic success and failure of private sector firms, scant use has been made so far of such data for labour market policy evaluations.

Retrospective Studies

Panel data do, however, have a number of drawbacks. Since repeated measurement is a constituent component of each panel study, it is obvious that not all persons who have participated in the first round of interviewing are also available in subsequent interview sessions. Statistically this problem of drop-out from the panel population is called attrition. Longer running panel studies have to deal with the loss of cases as the study evolves. When panel attrition is correlated with programme participation, this source of selectivity bias has to be dealt with in order to arrive at reliable estimates of programme effects on labour market processes like wage attainment.

Still less common is the use of retrospective designs (although more common in sociology and demography) which rely on the capability of interviewees to recall events with sufficient accuracy. They constitute a means of retrieving longitudinal data in one single interview session. Whereas a retrospective study focusing on qualitative information can use a biographical approach, studies focusing on quantitative information based on representative samples use standardised forms of questionnaires during interviews applying a event-

20

1

1

I

Outlook

oriented design of questionnaires. The duration until a specific event (for example job loss) occurred is a major component of data collection, facilitating the application of longitudinal research methods.

studies based on Administrative Records

In most countries access to administrative records of, for example, social security contributions or pension contributions are severely restricted to public authorities dealing with these confidential data. The need to review from time to time the level of coverage of employees, examining possible sources of fraud, or the permanent monitoring of financial in- and outflows of these funds have lead to an accumulation of large bodies of data. There now more analytical evaluation attempts based on these records and they constitute a major source of unexploited data for evaluations of labour market policies.

It is worth noting that the major, but still under-exploited, potential of the different types of longitudinal designs lies in the combination of various types of longitudinal information. Since longitudinal studies are fairly expensive research endeavours, the sample size of studies tend to be kept at a minimal level, where it is still possible to make statements representative of the chosen reference group (a country, or a region, or a specific socio-economlc target group) However, this minimalist strategy has had negative consequences on the precision of estimated effects of labour market interventions. One possible strategy to reduce error margins associated with estimated effects of labour market programmes is, of course, to collect larger sets of data either by increasing the original size of samples or by allowing longer periods of observation which also increases the number of observations of, for example, completed spells of unemployment.

There are still very few long-run longitudinal studies which combine individual-level information with aggregate national level indicators. In order to improve targeting of labour market programmes, more detailed knowledge about a personis labour market experience and other parts of the life course is necessary to better disclose unmeasured effects of unobserved heterogeneity or self-selection. Hence, either by way of retrospective data collection or by prospective follow-up of samples of participants and control groups, the path dependency of events and interventions is a non-negligable determinant of programme success or failure.

1 The Case of Luxembourg

Presentation of Best Practice

Uwe Warner: The reason why 11mhere is that my centre, the CEPS, is strongly connected to the implementation, the monitoring and the evaluation of the 4 EC programs in the fight against poverty. We started as Luxembourg partner of the European Commission in the evaluation around the first program. There the main problem was to get information on the economic situation. To monitor the economic situation of the country with special attention to poverty. The result of this work was that my colleagues needed to explain the dynamics, so the next step was a panel study. They created a household panel in 1985. It has 2,000 households, 6,000 individuals, and about 800 variables on the individual level.

It informs the researcher on demographics, on the activities on the labour market, on education, participation in occupational training, on the special income situation, on the social security system, female labour force, female participation in education. We do not deal with unemployment because in Luxembourg unemployment is not a problem.

Instead of that we have another problem, the .transfrontaüer", workers from neighbouring countries commuting into Luxembourg. These are people living in neighbour countries like Germany, France or Belgium and coming across the border to Luxembourg to work there. The third step was strongly related to that.

There was the need to compare social security policies, LMPs of our neighbour countries to our own country. So we built a data base which includes the household panels from Germany, Belgium, France, the UK and the American Panel of Income Dynamics.

In building up a comparative panel data base we encountered 2 problems:

how to harmonise the information from the different national panel studies and how to standardise the indicators. We did it in two ways. We made a distinction between harmonisation and standardisation. Harmonisation means that we calculate variables in such a way that these variables in every country deal with the same social fact. If I want to do a comparison, I need for all countries the same measurement, so we standardised information and variables. The most famous standardised variables are the International Standard Classification of Occupation. So we added to each national panel of households the ISCO, which is an indicator that gives me information about the position and the activity on the LM, and which is standardised in a way so that I can compare directly the effects on the LM. The second we added was the International Standard Classification of Education, which made it possible to compare the effects of the educational system across countries.

22

'{

,5.

This comparative data set on household panels has about 600 individual variables and also aggregated some 200 variables on the level of households.

The data are free for use of academics, so anyone doing research can access the data. I insist on that because it's not true for a lot of data of EUROSTAT; it's very difficult to get access. An example is the labour force study. From there we saw that a lot of questions arose from the panel studies on households that we cannot answer because we have no information on the other side of the LM. So in 1986 Luxembourg started a panel of firms, of small- and medium-income enterprises. Since then we have 400 industrial enterprises every year. The structure of the panel is that we ask a frame of questions every year, and we add an additional module every few years. These modules inform on innovation, on education, occupational training and professional training, and about 4 more. It was the same procedure with the households. We built comparative data for these existing enterprise panels. We have data from

southern Denmark, northern France, Lorraine, and Belgium. So we have a very

ⁱ

good overview of what happened on the LM and the social structure of

households and household positions.

Next I shortly describe the European Household Panel Study. This is a

study conducted by the EC, EUROSTAT, which is now in the third wave of

panel studies on households, but with a difference to our comparative panel in

Luxembourg. The difference is that they pre-defined how to ask for the

information, how to measure. They defined exactly the questionnaire, and

major indicators, the information. They also translated the questionnaire. So

their target was to have exactly the same study in all European countries.

2 The Case of Germany

Jürgen Kühl: As you all know, there is a large, lasting employment deficit in almost all OEeD countries, 35 million decent jobs are lacking over years, so we have to concentrate on the local units of production (establishment) that create, that fill, that reconstruct, that destroy and even export jobs. We have to look at the demand side of the labour market which is under the current conditions the most important place of creating and filling jobs. And if you want to look at the impact of policy measures at firm strategies, at subsidies or whatever, it is not enough just to have cross-section data, but you have to trace this local unit of . production over time. This is a panel approach. We tried to construct such a

panel on establishments.

First we looked at the whole entity of establishments in Germany over 11 years and there are two findings: first, the average size of firms is just 14 employees per firm, which is very tiny; and second, that 82% of total employment is concentrated in firms which last over a long period of time. So, there are employment dynamics from new firms and from firms disappearing, but they are very tiny in comparison to the average size, and they are very tiny in comparison to those long-lasting firms employing on average 24 employees per firm, so the majority of employment and employment dynamics is concentrated on firms which exist over a long period of time, so it is worthwhile to introduce a panel on firms, a yearly survey on those long-lasting firms. If you have large employment dynamics, firm dynamics with 30-40% of firms being created each year and being destroyed each year it's nonsense to create a panel. But if you have a very stable core of more than 80% of total employment in the long run, it is worthwhile to trace these individual firms over the long term.

We had 2.4 million firms employing at least 1 employee covered by social security during 1977-87, so we tried to take those firms as a base for a panel study of firms.

The panel covers so far just west Germany, in 1996 it will be extended to eastern Germany, too. All firms of all economic sectors are included. Most panels concentrate on manufacturing firms, on surviving firms; we tried to cover all those public and private sectors and all business and service sectors, too, because the employment dynamics are very large and we cover all firms and all firm sizes, so those tiny ones or small- and medium-sized enterprises, they are very important. I mentioned already that the definition of observation is the local unit of production and the statistical basis, the addresses of firms, come from the employment statistics register, which covers everybody who is covered by social security, which is 80% of total employment. We run annual face-to-face interviews, we started in 1993, so in 1996 we will have the fourth wave of interviews.

j

]

.! :¡ :i

I

tl

~ fi

J'i

24

l

J

.~

j

l'l

¡

r