FS I 89 - 13
Evaluations of Training Programs:
- experiences and proposals for future
research
Anders Bjdrklund*
December 1989 ISSN Nr. 0722-673X
discussion papers
Swedish Institute for Social Research
University of Stockholm S - 106 91 Stockholm
and
The Industrial Institute for Economic and Social Research (lUI)
Box 5501
S - 114 85 Stockholm
Forschungsschwerpunkt
Arbeltsaarkt und
Beschaftlgung (IINV)
Research Unit Labour Market and Ejaploywnt (IIM)
Labour Market and Employment (IIM) Reichpietsch-Ufer 50
1000 Berlin 30
Abstract: Evaluations of Training Programs - Experiences and Pro posals for Future Research
In the Anglo-Saxon labor economics literature, there is a tradition to evaluate the effects of labor market training programs by compar ing the post-program labor market outcome between program parti cipants and a suitable selected group of non-participants. Statis tical regression analysis is used to control for the differences be tween the two groups which are not attributable to the program.
In spite of rather impressive development of new statistical tech niques and ambitious data collection efforts, the experiences of applied evaluation research (from the USA and from Sweden) are nega tive because the uncertainty about the true program effects is con
siderable.
Classical experiments where prospective program participants are randomly assigned to experimental and control groups are examined as alternatives to the non-experimental approach based on statistical techniques. The experiences from seven experiments (six from the USA and one from Sweden) are presented. The main conclusions are that classical experiments in general are feasible when new programs and changes in ongoing programs are considered. In situations when there is rationing of willing participants by the program authorities, ex periments can also be used. In quite many situations, however, the non-experimental statistical methods have to be used. Some possibili ties to improve such studies are also presented.
beitsmarktpolitischen Programmen (insbesondere Programmen der beruf- lichen Weiterbildung) in den USA und Schweden. Sie konzentriert sich auf die Frage, ob die Arbeitsmarktchancen der einzelnen Prograirantei1- nehmer durch die Programme mittelfristig verbessert werden, und sie untersucht vor allem die methodischen Probleme, die bei der Analyse dieser Frage entstehen.
Die Studie unterscheidet im wesentlichen zwei Typen von Wirkungsana- lysen; nicht-experimentelle Studien, die den Verbleib der Programm- teilnehmer mit dem Karriereverlauf anderer Personen vergleichen und statistische Regressionsanalysen verwenden, um die unterschiedliche Struktur beider Gruppen zu kontrollieren; und "klassische" Experimen- te, in denen potentielle Programmtei1nehmer von vornherein nach dem Zufallsprinzip in eine Teilnehmer- und eine Kontrollgruppe aufgeteilt und anschlieBend verglichen werden.
Die Bestandsaufnahme zeigt, daB nicht-experimentelle Studien trotz
verbesserter statistisober Techniken und neu erschlossener Daten nach wie vor unbefriedigend sind, da ihre Ergebnisse erhebliche Unsicher- heiten und Fehlermargen aufweisen. Experimente bieten vor allem dann eine Alternative, wenn arbeitsmarktpolitische Programme neu einge-
flihrt Oder verdndert werden und wenn die Zahl der Interessenten die
Zahl der vorhandenen Teilnahmeplatze Ubersteigt. In vielen Fallen koiranen allerdings auch experimentelle Methoden nicht in Frage. Wie in diesen Fallen nicht-experimentelle Analysen methodisch verbessert werden kbnnen, wird abschlieBend gezeigt.
- 3 -
TABLE OF CONTENTS
I Introduction
II Experiences of non-experimental studies
II. 1 American studies II.2 Swedish studies
m Experiments within evaluation research
III.l General problems with experiments
111.2 The National Supported Work Demonstration in the
USA
111.3 Wage vouchers for welfare recipients in Dayton, Ohio,
USA
111.4 Bonuses to the unemployed in Illinois, USA
111.5 The Employment Opportunity Pilot Project in the USA 111.6 The JTPA experiment in the USA
111.7 The Workfare experiments in the USA
111.8 Intensified job assistance service, Eskilstuna, Sweden IV Evaluation of evaluations with the help of experiments
V Macroexperiments
VI Suggestions for future evaluation research
VLl Experiments
VI.2 Data bases for non-experimental evaluation
Appendices
References
I Introduction!
Many of the labor market policy measures are intended to improve the individual's position on the labor market. The perhaps best example within the Swedish labor market policy is the vocational training courses for the unemployed. One usually says that it is a measure targeted to the individual.
It is not always clear from the formulations of the policy goals how "position on the labor market" is to be operationalized. One measurement which is used often is the income from gmnful employment during a certain period, for example one year. This measurement contains a series of components: the number of days of employment during the year, the number of hours worked and average hourly wages influence the total income within the given year.
Whether yearly income or any component of it is the most relevant variable is hard to say in general. It is likely to vary between different measures of policy and between different institutional characteristics of the labor market.
Most probably, however, many politicians emphasize the fact that the number of days of employment increase (and the number of days of unemployment decrease) as a result of political efforts, but even increased hourly wages would undoubtedly be considered by most as an achievement.
Even an improved working environment can be included in the aims for vocational training courses for the unemployed.
Whatever variable that is most appropriate in a specific institutional setting, it is an important task for evaluation research to be able to quantify the effects of labor market policy measures. The dominant research in the field of
labor economics in the last decades has been built on some form of
comparison of the incomes after the training period between participants and non-participants. A properly constructed comparison should indicate how large the effect of the measure is. The principal methodological problem has thus been to make this comparison in a constructive manner.
!I am grateful to a referee for useful and thoughtful comments on a previous
draft of the paper.
5 -
Most studies which have appeared in the international journals have been American in general and from the USA in particular. The Swedish evaluation research has been strongly influenced by the American studies.
Two different approaches have dominated the literature. The one is classical experiments, which means that prospective program participants have been randomly divided into an experimental group, which may take part in the program, and a control group, which may not. As all the factors which affect the outcome (in terms of annual income or something similar), with the exception of the participation itself, must be equal in both the groups, the difference should be due to the program itself. Sometimes randomized
experiments are characterized as the ideal method.
The other approach uses the data from actual program participants and non-participants. These individuals have been chosen in the selection process which decides participation in the program under study. In practice this selection proems most often concerns decisions both by the authorities who are responsible for the measures and the prospective participants themselves.
Various statistical methods have been developed for allowing conclusions to be drawn about the effects from data on non-randomly selected groups as well. "Matching techniques", which mean that the participants are compared with a group of non-participants with similar characteristics —a comparison group2 —is one of the methods. The most usual one, however, has been to
"control for" systematic differences between participants and non-participants with the help of some form of statistical regression analysis.
Significant methodological development within the non-experimental research approach has taken place in the past 10—15 years. Overviews of this literature can be found (in Swedish) in Bjorklund (1983) and (more detailed and technical) in Heckman and Robb (1985a, 1985b). In the USA, significant efforts have also been made to produce better data bases, which can be used for the non-experimental methods.
2The term comparison group will be used about a non-randomly selected group of non-participants, whereas the term control group will be used about randomly selected group.
Despite these ambitious efforts made in both statistical method and construction of data bases, many researchers still house serious doubts today about the possibilities of producing reliable estimates of the effects of the program with the help of non-experimental techniques. In an article which was published in the Evaluation Review, Heckman, Hotz and Dabos (1987) characterize the frames of mind in the following way:
In the past three years, a pessimistic mood has become dominant in the evaluation literature. New studies have been put forth that claim the non-experimental methods of program evaluation produce "unbelievable", "sensitive", "nonrobust" estimates of program impacts and that randomized experiments are necessary to produce reliable estimates.
The "problem of comparison" which the research mentioned above focused on is not, however, the only problem which should be taken into account when one tries to quantify the effects of programs like vocational training courses for the unemployed. An (often implicit) prerequisite for the experimental as well as the non-experimental approach is that the effects are limited to those who really participate in the program. It is thus assumed that those who do not participate in vocational training course for the unemployed are not affected at all by the fact that others do so. This assumption is far from being incontrovertible. One can quite easily imagine both positive and negative effects on other groups. Such indirect effects can arise both during training and afterwards. This problem was emphasized in the first Swedish evaluation studies and in EFA's report "Att utvardera arbetsmarknadspolitik" (To evaluate labor market policy) from 1974 (SOU 1974:29). No method for quantifying such effects could be suggested and in the international (primarily American) evaluation research which is published in journals, the problem is normally not even taken up. This implies that the indirect effects are presumed to be zero, or at least small.
The purpose of this paper is to discuss the possibilities of progressing within the evaluation research which aims at quantifying the effects of measures such as vocational training courses for the unemployed. The presentation is influenced by the existing literature, which is directed towards solving the so
- 7 -
called comparison problem. Both the experimental and the non-experimental approaches will be taken up.
A summary of the r^ults from non-experimental studies of American and Swedish training programs in Section II comprises the background. It is the uncert^nty and the low precision in these studies which lies behind the desire to conduct classical experiments.
After this there is a long section presenting six different experiments which have been conducted, and one more to be carried out in the future, in the area of labor market policy. Even though only one of these concerns vocational training courses for the unemployed, the methodological experiences are relevant for the majority of individually-targeted measures.
Experiments can also be employed to evaluate non-experimental studies - a kind of "evaluation of evaluations". This approach - and the studies which make use of this possibility —is presented in Section IV.
The next section contains a discussion of the so-called "macroexperiments", in which towns or regions are randomly distributed between experimental and control groups. Otherwise the most common experiments have been
"microexperiments", that is, individuals have been randomly assigned to experimental and control groups.
The study ends with a section which discusses how experimental and non-experimental studies can be developed separately and in combination.
Some concrete suggestions for future evaluations research are presented.
n E3q)erieiices of non-experimental studies
n.l American studies
Since the beginning of the 1960's it has been characteristic in American labor market policy to test a wide variety of labor market policy programs, usually on a small scale and during relatively short periods. Another distinguishing trait is that the different programs have been subjected to evaluation studies of various kinds, which explains the existence of hundreds of studies.
Regarding methodological experiences, the studies around the CETA program
during the 1970's are of greatest interest —which is why we confine ourselvesto a treatment of them alone.
CETA (Comprehensive Employment and Training Act) became effective in
1973 and contained different subprograms such as "public service employment", "work experience" and "direct referrals" (a pure job assistanceservice). Some participated in several of these activities. The Department of
Labor took the initiative to collect data for future evaluations: the data base
was called the "Continuous Longitudinal Manpower Survey" or CLMS. First of all information was collected about those persons who participated in
CETA during 1975 (6,700 individuals in the sample) and 1976 (13,300
individuals in the sample). Variables describing among other thingsdemographic characteristics, family status and labor market experiences for a
period of four years beginning with the year before the start of the program were included in the CLMS. Data from a sample of persons from the American labor force survey (from March 1976) have been added to this information about the CETA participants. Register data about incomes("pension-based" incomes from the Social Security Administration) for the
period 1951—1978, that is, two years after the participation in the program and many years before, were gathered for both groups.Even if the resulting data base is the most ambitious attempt to create an evaluation data base, it does have certain flaws. Some primary variables such as certain labor market experiences, union membership and region are missing (that is, they do not exist for all the individuals) and the measure of income
- 9 -
which comprises the main outcome variable also has certain flaws. The greatest of these, however, is that it is not possible to determine which of the people included in the comparison group have also participated in CETA. If this had been possible — and it would have been with better register information —these individuals would naturally have been deleted from the comparison group. The analysis which has been made with the existing control group is therefore not completely reliable: in general the effects are most likely somewhat underestimated due to the so—called contamination bias which arises (see Bassi (1983) for a more in depth discussion).
A great many studies have been based on the data which were collected in this way. The reason for the large number of studies is undoubtedly due in part to the fact that the material was public—use data and was available to the entire r^earch community at a low cost.3 Barnow (1987) contains a careful methodological examination of these studies and a summary of the
results. An accessible overview of the results of the studies from both the
CETA program and other American programs is found in Bassi and Ashenfelter (1986). Characteristic for all these studies is that they make use of the possibilities that longitudinal data or panel data offer. The outcome variable employed is annual incomes (during 1977 or 1978), and a comparable variable is also available for periods of time prior to the program, thus making it possible to "control for" the level of the incomes as well as their rate of change before the program. This can be accomplished partly with the help of matching techniques, which mean that the participant's results are compared with those of a group with similar characteristics including among other things previous incomes, partly with the aid of regression techniques where prior incomes are employed as control variables. In practice a
combination of these methods have been used in those studies which Barnow
summarizes. First a number of individuals from the complete control group, which was a random sample from the entire population, was put into a more limited control group. Then regression analysis was applied.
Within applied labor economics research, many models with appurtenant
3In a form which guaranteed that individuals in the material could not be
identified.
estimation methods have been developed in order to estimate the effects of
vocational training courses for the imemployed based on longitudinal data.
Each model can be said to have two parts: one which describes the dynamic structure of the incomes (or the development over time), and one which describes how the selection into the program occurred. The choice of models depends therefore on how both of these parts and the relation between them should be specified. Heckman and Robb (1985a and 1985b) contains a set of models and appurtenant estimation techniques which can be applied.
Despite the relatively good data base and the new sophisticated methods which have been developed and put into use, the results are far from unequivocal. The existing element of uncertainty is of two types. First, every
estimated model gives an ^timate of the program's effect which is connected
with uncertainty (the sampling error of the estimate) Second, differentmodels give different results and it is difficult to determine which model is
most reliable. Both the sources of uncertainty are illustrated by some of the results from two of the studies which Barnow examined, see Table II.1. It must be stressed, however, that neither Barnow nor others explicitly emphasize the first type of imcertainty. This is of course the classical uncertainty which every estimate is associated with, and which determines the width of the confidence interval for the parameter. My impression from the literature is that the standard errors are often very high. Despite this it is unusual to have explicit discussions concerning different strategies for reducing this uncertainty. In Bjdrklund (1989), Chapter 9, 1 give some possibilities.The first type of uncertainty is indicated by the fact that every point estimate is completed with information which states plus/minus two standard errors: 740 ± 260 thus means that the point estimate is 740 dollars, but a 95%
confidence interval is 480 —1,000 dollars. The narrowest confidence interval in the table (which was obtained by Bassi for "fixed effect model with series-correlated random term" for 1977) is 2 times 192 dollars, that is , almost 400 dollars. Considering that the incomes that year were between 3,500 and 4,700 dollars for thrae persons, this uncertainty equals around ten percent of the incomes, which is very high. For other groups and methods, the confidence interval can have a width of 1,500 dollars, that is, around
11 -
Table n.l Some results from two studies of the CETA program
A. Effect (in dollars) on annual incomes in 1977 and 1978 according to Bassi (1984).
Method White women
Grouo of narticioants Women from
minority group
Men from
minority group
Fixed effect model Incomes 1977 Incomes 1978
740 ±260 1 108 ±336
426 ±470 531 ±570
117 ±632 27 ±772
Fixed effect model with series-correlated random term
Incomes 1977 Incomes 1978
987 ±192 1 621 ±220
— 164 ±426
132 ±538
Creamine model Incomes 1977
Incomes 1978
:
626 ±326947 ±388 271 ±430240 ±540B. Effect (in dollars) on the annual incomes in 1978 according to Dickinson, Johnson, West (1986)
Method
Grouo of oarticinants
Adult men Adult women
State dependence
model -690 ±278 13 ±232
The studies are presented in more detail in Appendix A.
one-third of the income level.
The next type of uncertjunty is a result of the fact that different methods (and different studies) yield different results. The ^timates which are presented from Bassi's study have all been exposed to a test procedure'', and none of the models could be discarded. This means that it is impossible to claim that one method is more reliable than any other method. For the group white women, the results vary between 740 and 987, and 1 108 and 1 621 (1977's and 1978's incomes, respectively). For women from minority groups - the corresponding variation are 426 to 626 (1977's income),and 531 to 947.
For men, the variations in the point estimates are less, but on the other hand, the confidence intervals are exceptionally large.
The uncertainty resulting from differences in method is increased if Bassi's results are compared with those obtained by Dickinson, Johnson and West.
For adult men, they got relatively strong negative effects, which furthermore were statistically significantly different from zero, while the effects for women were close to zero. This is the study which provides the worst results of all the ones presented by Barnow. To be sure, these two studies are based on different subgroups from the CLMS material, which means that they are not really comparable, and thus the results might be compatible with each other.
Barnow judges however that the effects which Dickinson, Johnson and West received are far too low due to different faults in the material (see Barnow:
p. 189). For this reason his general opinion is that most of the studies get
effects in the range from 200 to 600 dollars (that is, 2.5%—7.5%), somewhat
higher for women and somewhat lower for men. The uncertainty is of importance, however, for the reasons given above. Barnow is of the opinion that classical experiments based on random selection procedures "..appears to be the only method available at this time to overcome the limitations of nonexperimental evaluations."Bassi and Ashenfelter (1986) in their summary of research are of the same opinion about the uncertainty in the results and the need for experiments. It
''This test procedure is reminiscent of that which is presented in Section IV
below.
- 1 3 -
is these conclusions that Heckman, Hotz and Dabos (1987) are referring to when they claim that a pessimistic mood prevails within the evaluation literature (cf. the quote above in section I).
n.2 Swedish Studies
Three studies of vocational training courses for the unemployed in Sweden were conducted in the beginning of the 1970's. All of them indicated that the incomes of the participants increased as a result of the training they received.
From a statistical point of view, however, it is unclear how precise these estimates are. It is primarily for two other reasons, though, that the studies have a limited value today. In the first place, all the studies were based on the vocational training courses for the unemployed conducted in the 1960's, when the level was considerably lower than today's. If there are diminishing returns, which is a reasonable assumption to make, then the effects may be lower today. Secondly, by today's criteria the methods used for evaluating the effects are not satisfactory: the studies were made before the new insight into methodological issues were made in the 1970's.
In Bjorklund (1989) I have tried myself to apply some of the methods which were published in the American literature. The study has the advantage of being able to make use of longitudinal data, but the material otherwise is not primarily constructed for an analysis of labor market policy. The sample is rather small, among other things.
The results unfortunately indicate the same type, and approximately the same degree, of uncertainty as in the American studies. Table II.2 indicates a selection of the results. Each individual estimate of the program shows an uncertainty corresponding to a confidence interval of a length between about 10 and 38 percent units. (Note that the effects in the table are expressed in relative terms, because log income and log wage were used as dependent variables.)
Added to this uncertainty is the fact that there is a disturbing variation in the results for the two different methods. This is especially true for the effects on hourly wages and on annual incomes.
TableII.2 Some results from a study of the eSects of the Swedish vocational training courses for the unemployed
A The effect of vocational training courses for the unemployed in 1976-80 on employment and hourly wages in 1981
State dependence Fixed effect
model model
Effect on the percentage with employment when
interviewed in 1981. 0.055 ±0.060 0.080 ±0.076
Relative effect on
hourly wage for those
with employment, -0.043 ±0.054 0.065 ±0,076
B. The effect of vocational training courses for the unemployed in 1976-1982 (first 6 months) on income during 1983.
State dependence Fixed effect
model model
Effect on the percentage
with income at some time
during the year. 0.009 ±0.048 0.013 ±0.068
Relative effect on the annual income for those
with incomes during
the year. 0.044 ±0.16 0.186 ±0.190
Source: Bjdrklund (1989)
The methods are described in more detail in Appendix B.
- 1 5 -
m Experiments within evaluation research
The purpose behind a classical experiment is to solve the comparability problem witch the non-experimental approach suffers from. Classical experiments do, however, suffer from problem of their own. There are extensive discussions in the literature concerning just such problems (see eg.
Burtless and Orr, 1986). These problems can best be understood if they are discussed in connection with concrete examples, either experiments which have been carried out already or concrete suggestions for new experiments.
The main purpose of this section is therefore to present a number of experiments which have been conducted and to discuss their specific problems. Before we begin, however, we will give a short presentation of some general problems which experiments are said to be connected with.
III.l General problems with experiments The representativeness of the experimental group
In order for an experiment to yield convincing results, it is mandatory that the experimental group is representative of the group which will participate when the program is operating normally. One can distinguish between two pure cases here. Assume first that (in principle) everyone in the target population who wishes to participate in a program may do so. One example is the mobility grant within Swedish labor market policy which all unemployed persons are entitled to. Even the courses within vocational training courses for the unemployed have been available to a large extent to most of those applying during the 1970's and 1980's in Sweden. Assume further that for the purposes of evaluation, those who have applied to participate in the programs are randomly assigned to an experimental group, which is permitted to participate (or benefit from a service) and a control group, which is not permitted to participate. In order to assure that this division is reliable, the existence of the experiment itself must not affect the composition of those who apply to participate. If the uncertainty which is introduced by the experiment affects who applies, the result might be misleading.
Another case concerns a program in which only a part of the applicants normally are given the opportunity to participate. If, for example, the number of job assistance advisors is insufficient, an employment exchange office is normally forced to set up priorities among those seeking work so that some receive relatively extensive help while others get only limited service.
An experiment aimed at measuring the effects of job assistance service can thus lead to a situation in which a randomly selected experimental group who has received services is different from the group who would have received services had the program been operating normally. This, too, might produce misleading results.
These examples show that a randomly selected experimental group is not necessarily representative of those who participate when the program functions normally and the decisions concerning participation are made by program authorities and perspective program participants.
The Hawthorne effect
The concept "Hawthorne effect" originates from a study at the Western Electric Company in Chicago during the 1920's. The effect of changes in the working environment on labor productivity was examined. The first studies showed that even a clear worsening of the working environment led to increased productivity.^ The reason was that the employees appreciated the experiment itself and, as a direct consequence, productivity rose. Since then the concept Hawthorne-effect has been used for an effect that is caused by the experiment itself rather than the program which is studied by means of the
experiment.
When making experiments on labor market policy, one must ask oneself if the special circumstances occasioned by the experiment lead to the agents involved acting in a different way. By "involved agents" are meant both the program participants themselves and those who are behind and administer the programs.
sLater these paradoxal results have been questioned (see Franke and Haul (1978)).
- 1 7 -
First of all, there is a chance that the administrative authorities can act in a different way. One example might be that the object of the experiment is something new and that people do not have the time to learn how to use it effectively during the period of the experiment. Another example can be that good results from the experiment can lead to additional grants to the authorities and that the employees therefore exert themselves more than usual to achieve positive results. In both of these examples, one can believe that the risks of getting misleading results are greatest if the experiment is of very short duration.
In conclusion, it is important to insure that all the agents involved act as
"normally" as possible during the course of the experiment.
Indirect effects
Certain labor market policy measures can lead to improvements on the labor market for the participants at the expense of others. Job assistance service, training, and various wage subsidies can have such consequences. An experimental design for a study can therefore yield results which indicate great improvements for the experimental group as compared with the control group, but this difference could merely reflect the fact that one group got the existing jobs at the expense of the others.
One concrete form of such effects is the so-called "queuing bias". If the program slots generally are distributed according to a certain queue system, the experiment can lead to the experimental group's coming first in line while the control group ends up last. In such a case the results would appear to be very positive without creating any effect on the total employment.
Even positive, indirect effects are possible, however. Training and moving allowances can lead to the reduction of bottlenecks in production being reduced and as a result, to other group's getting employment. Training can also have positive external effects; the knowledge which an individual acquires from a course can be passed on to others as a normal part of their
work.
Another type of indirect effect created by the experiment can work via traditional market mechanisms. If the experimental group is given a certain type of professional training, the increased supply of this labor category can lead to wage changes both for this group and for others. The direction of such effects — that is, if the rest of the individuals win or lose — is generally
considered difficult to determine.
Indirect effects on others than those in the experimental group thus create problems for this methodological approach. It is important to emphasize however that these problems are not peculiar to experiments as such, but are equally valid for the non-experimental approach.
The conclusion however is that every study based on an experimental design must be supplemented with a judgment of to what extent indirect effects of various types can have occurred, and if so, their sphere of influence.
Ethical problems
An experimental study, with random assignment of individuals into experimental and control groups, is considered sometimes to be impossible for ethical reasons. For this reason, many perhaps reject the idea of carrying out experiments without detailed analyses of how these ethical problems actually appear and how they relate to the advantage of the experiment. Since a number of experiments within labor market policy have actually been conducted, one cannot simply discard the idea by referring to some diffuse, ethical problems. One of these experiments was conducted in Sweden in the beginning of the 1970's (see below, section III.8).
In general, the main problem is that it can be ethically objectionable to deny certain persons assigned to the control group the type of service which is subject to an evaluation. The degree of severity varies from case to case. If it is a question of a right which has existed for a long time and is strongly rooted in the society and perhaps even stated in the law, the program can naturally appear ethically impossible to experiment with. For example, denying unemployment compensation to a randomly selected percentage of the unemployed for the purpose of studying the effects of the compensation
- 1 9 -
system on the length of the unemployment periods is undoubtedly unacceptable to most. On the other hand, it might be possible to accept random distribution concerning more limited, new types of policies.
It should be emphasized as well that it can be unethical to refrain from experiments. If such a study is the only realistic way of acquiring knowledge about the effects of a costly measure, it can appear unethical to refuse to undertake a study which can inform the Labor Department, the Parliament and the tax payers about its effects and profitability.
Costs
One practical problem in connection with experiments can be high costs. Just as with the ethical problems, it is difficult to make a general statement about the problem of cost. Experiments can be made in different ways, and the direct costs incurred must also be weighed against the potential gains in the form of improved labor market policy.
in.2 The National Supported Work Demonstration, USA ^
The National Supported Work Demonstration, which was conducted in 15 different places in the USA between 1975 and 1978, is the most well—known and ambitious experiment in the field of labor market policy. It is documented in a book (Hollister, Kemper and Maynard (1984)), it has been the source of many articles (among others a special feature issue in the Journal of Human Resources, Fall, 1981) and it has been employed in the
"evaluations of evaluations" carried out in the middle of the 1980's, presented
in section IV below.
The policy instrument tested in this experiment was directed to severely hard-to-employ groups. Four target groups were defined:
6This presentation is based primarily on Hollister, Kemper and Maynard (1984).
1 Women with AFDC (Aid for Families with Dependent Children). The requirement was that the women received AFDC and had done so for at least 30 of the preceding 36 months. Further, the youngest child must be 6 years
old or more.
2 Addicts. They should be at least 18 years old and be participants in a treatment program or have participated in one during the last 6 months.
3 Ex-convicts. They should be at least 18 years old and have been
incarcerated for a crime within the last 6 months.
4 Youths. They should be between 17 and 20 years old, have dropped out from high school and not been in school during the last 6 months. For at least 50 per cent of the youths it was required that they had participated in some sort of criminal activity.
All the groups had to be unemployed and to have worked less than 3 of the
last 6 months.
The labor market policy instrument which was tested on these groups was a type of sheltered or half-sheltered employment with four characteristics.
First, the work was limited to 12 or 18 months. Naturally, the employees could voluntarily leave their jobs earlier and they could be fired if they deserved it, but it was clearly stated that the job was over after 12 (or 18) months. Second, the work was carried out in groups in which the participants had similar backgrounds. The idea behind this was that the participants would be able to support each other both by understanding each other's problems and by setting examples for each other. Third, the work teams were supported by staffs who were knowledgeable both in the work area and in the field of the participants' special problems. Fourth, the demands on the work done by the participants were increasingly heightened, so that they would ultimately equal the requirements existing on the open labor market. One instrument to achieve this aim was a special wage and bonus system. The basic wage was not permitted to be under the legal minimum wage, but was somewhat below the monthly wages for local blue-collar workers. The bonus was depending on individual productivity, and since the demands were
-21 -
heightened it became gradually more difficult to receive bonus.
This type of labor market policy instrument was enacted in 15 locations spread all over the USA. In 10 of them, the use of random selection was employed, that is, classical experiments. In each place, extensive work was required to get the project to work. To start with, close cooperation with the authorities able do direct participants to the program was necessary. These authorities included welfare offices, rehabilitation centers for addicts and alcoholics, and prison authorities. Further it was required that the potential participants be randomly assigned to experimental and control groups. More than everything else, it was necessary to find places of employment where the participants could work in groups and leaders who had the necessary qualifications.
Finally, the evaluation demanded a large data collection. The bulk of information about income and employment development after the participants had left the program was obtained through interviews, which were supplemented with some register data, primarily to get a better measurement of occurrences such as continued criminality and addiction.
(The interview method led in addition to some problems of non-response, but a closer analysis indicated that they were most likely of less consequence.) Furthermore, the activity in the various places had to be coordinated so that a certain degree of uniformity could be achieved. Even funding made demands on coordination. Different federal departments and the Ford Foundation contributed, but a large part of the local program activities were financed locally. Thus, it was a very large project, requiring an extensive amount of work in the form of coordination. The MDRC (Manpower Demonstration Research Corporation) was created as a coordinating organ. The gradual accumulation of competence and experience during the course of the project was preserved by making the MDRC a permanent organization. It has headed many evaluation projects since then.
The experimental planning on a more statistical basis meant that 1620 women with AFDC, 1154 addicts, 1458 ex-eonvicts and 1252 youths were selected. With the exception of one town, half of them were placed in the
experimental group and half in the control group. It is important to state precisely at what point this random selection occurred. In this case it took place after a careful selection of both the authorities who directed applicants and those who were responsible for the program activities: the leaders were able to interview all the participants and reject those who were adjudged unsuitable. In other words, selection to the experimental and the control groups was made from a group who had themselves agreed to participate and who had been screened by the program administrators.
The random selection meant however that only about half of those who were considered suitable by the field staff were able to participate. The rest had to go into the control group. This created - see Hollister, Kemper and Maynard (1984:35) —a certain dissatisfaction, but the authors did not feel that it was serious enough to affect the results to any great extent.
Results
The studies of the effects of the program have focused on whether the participants were employed more and received higher incomes than they otherwise would have been/done, and whether criminal behavior and
addiction decreased.
Stated briefly, the major results are that women with AFDC who participated in the program had a better employment and income development than the control group, that is, a positive effect is evidenced.
Addicts had a lower arrest frequency. On the other hand, no effects on employment or incomes could by found for this group. Nor was the use of drugs affected for this group, despite the decrease of arrests. No effects on employment or incomes were found for ex-convicts either, but various information problems made the analysis of this group more difficult than for the others. Finally, no effects for the vouth group were found: the participants in the control group showed in general the same development as the experimental group.
Attempts were also made to carry out social cost/benefit calculations for the program. On the plus side, the value of what was produced during the project
- 2 3 -
and the value of the higher employment and lower crime rate after the project were included. On the minus side were included among other things the costs for the leaders, the premises, etc. The calculations, which contained several reservations, indicate social profitability for the measures directed towards the women with AFDC and addict groups, while the measures directed towards the youth group gave negative results. Regarding the measures aimed at the ex-convict group the result was very unclear.
Methodological limitations
The National Supported work Demonstration in all important aspects seems to be a well-conducted experiment. The choice of the experimental and control groups was made from a population which had gone through the selections made by prospective participants (enrolling into the program) and
by the authorities (choosing the ones who are most suitable to participate).
Therefore the experimental group is certainly representative for those who would have participated if the program had been operating normally.
A certain risk for the Hawthorne effect could have existed. The program which was introduced and tested was namely new, and the field staff can therefore have been unused to handling these policy instruments. One can also imagine that the program could have been improved if the evaluation results could have been applied in practice.
Indirect effects on different labor markets, the control group included, might have occurred and might also have slanted the results. It is however impossible to say anything more definite on this matter.
III.3 Wage vouchers for welfare redpients in Dayton, Ohio, USA^
The background of this project is that the US Congress in connection with a re-organization of the CETA program in 1978 directed the Department of Labor to carry out a study of direct wage vouchers. A wage voucher is a 7This presentation is based on Burtless (1985).
certificate given to individual job seekers, entitling any employer who hires the seeker to a certain wage subsidy. Congress directed the Department of Labor to test the instrument on groups of job seekers who were entitled to other type of service within the CETA program.
The project was conducted jointly with Mathematica Policy Research, a research company. The design of the experiment was very simple. Between December, 1980 and May 1981, 916 participants, all welfare recipients, were selected: just under half received AFDC and the rest got general assistance payments. They were randomly divided into three groups. One group was given a "tax credit voucher", which meant that the employer who hired the job seeker was entitled to a tax reduction. A second group got a "direct cash rebate subsidy", which meant that the employer who hired the job seeker was entitled to a direct subsidy. The subsidy amount for both groups was 50% of the wage bill for the first year (with a maximum-limit of $3000 for subsidies) and 25% of the wage bill the second year (with a maximum limit of $1500 for subsidies). A third group received no voucher at all and thus constituted the control group.
All three groups participated in a two-week—long course directed towards teaching and training in how to seek employment, a sort of job-himting club.
Both experimental groups were also given information at some point in the course about how they could use their vouchers. The teachers at the courses circulated between the groups so that no systematic "teacher effects" could occur. In this way the authorities also lost the possibility of giving the experimental groups the best teachers so that the results would be as good as possible.
Results
The results from the experiment can be presented in a very simple way without any complicated statistical or econometric techniques. The fact is that Table III.l is the only documentation which was published about the results of the project. Burtless in another context (Burtless 1988) claimed that one of the advantages with experiments is that the results can be understood very easily even by those readers who lack knowledge of statistical
- 2 5 -
methods.
As is seen in the table, the results are worthy on notice, to say the least.
Those who received a benefit in the form of a wage voucher show undisputably worse results than those who belonged to the control group! The difference is clearly significant, which means that it is highly improbable that the difference between the groups can be explained as occurring by chance.
On the other hand, the results for both types of vouchers are more or less the same, which means that it makes no difference how the companies get the
support.
Table m.l Employment figures for the experiment and control groups in Dayton
Group Sample size Number placed Percentage
placed size in jobs
Tax credit
voucher 247 32 13,0 %
Direct rebate
subsidy 299 38 12,7 %
Control group 262 54 20,6 %
Total 808 124 15,3 %
Source: Burtless (1985)
Notes:
The employment numbers refer to whether work is obtained within an 8—week period after the course in job seeking.
The 808 of 916 original participants who are included in the results are those who completed the course in job seeking.
The question is how one can explain this paradox: that those who could offer the employers an economic compensation in the form of a voucher fared worse than the others. Some mechanism must not only have neutralized the economic advantages inherent in the wage vouchers but must also have outweighted them. Burtless' explanation is based on the idea that a wage voucher can also act as a signal for the employer. When the job applicant presents his voucher for a presumptive employer, he must also explain to him why just he is able to offer him one. The reason, of course, is that he is a welfare recipient. This means that the employer has received a type of information about the job seeker which he otherwise would not have received.
If this information is perceived negatively due to the fact that welfare recipients suffer more often from various social problems than others, a voucher can act as a warning signal. As Burtless wrote, the wage voucher can act as a signal that the seeker is "damaged goods".
Another result of the study is that only a smaller part —about one third —of the employers who in principle would have been entitled to the wage subsidies (70, according to the Table) in actuality demanded their money. One explanation can be that several of the job seekers with vouchers did not make
use of them. According to Burtless, there are informal reports from the staff
who conducted the experiment that certain job seekers deliberately did not use vouchers. This in turn can depend on the fact that they realized that the employers would interpret them as a negative signal.An experiment of this type does not lead primarily to methodological speculations but rather to thoughts of labor market policy. An important
question is how general can the result be: can similar mechanisms yield
negative effects even from other types of labor market policies like training, even for other countries? This possibility cannot be excluded, at least. Since wage subsidies occur quite extensively in many countries, this study must constitute a warning for many decision-makers in labor market policy issues.The best way to react to this warning naturally is to test the effectiveness of existing wage subsidies in other coimtries with the help of experiments.
Another opinion might be that wage subsidies ought to be used to compensate for visible and obvious handicaps rather than for invisible and less obvious
- 2 7 -
ones. If for example persons with visible physical handicaps offer wage vouchers, these will not necessarily be taken as a signal of invisible problems.
It is possible that wage subsidies can function properly when they are held by persons with visible and obvious handicaps but that they are received negatively when possessed by persons with invisible and less obvious handicaps.
in.4 Bonuses to the unemployed in Illinois, USA^
Due to long—term criticism of the system for unemployment insurance and dissatisfaction with the results of non-experimental studies of how unemployment compensation affects the job-seeking behavior of the unemployed, the Department of Employment Security in Illinois decided to examine how well bonus payments to the unemployed could work as instruments for reducing the length of unemployment. They chose to use a classical experiment for this investigation. The project was carried out in cooperation with the W.E. Upjohn Institute for Employment Research.
Two types of premiums to the unemployed were tested. In the first case someone who had just become unemployed and who was entitled to unemployment compensation was offered an extra bonus of $500 for himself if he found work within 11 weeks. Two requirements were made, however: that the job should be held for a minimum of 4 months and should consist of at least 30 working hours per week. The other premium meant that the unemployed person received a wage voucher worth $500 which he could give to an employer who hired him within 11 weeks. The job requirements were the same as in the first case with the bonus. The size of both the bonus and
the voucher was equivalent to about 5 per cent of an annual wage or four weeks unemployment compensation. The amount decided upon was determined among other things by the $750,000 budget available for premium payments to companies and the unemployed.
To be entitled to either an offer of a bonus or a voucher, it was required that
SThis presentation is based on Woodbury and Spiegelman (1987).
the individual (i) had started a period with unemployment compensation between July 29 and November 17, 1984, (ii) was entitled to 26 weeks' compensation, (iii) was registered at one of 22 different Job Service Offices in northern or central Illinois and (iv) was between 20 and 55 years of age.
Those who filled these conditions were divided randomly into three groups: an experimental group who was offered a bonus, an experimental group who was provided with vouchers and a control group who did not receive any extra offer beyond the regular unemployment compensation which naturally also went to both the experimental groups.
The information about incomes and contributions both before and after the
period of unemployment was gathered from various registers. In this way the non-response problem of the interview technique was avoided.
Results
The study sheds much light on the effects of introducing the extra incentives for intensified job seeking. Some of the results are collected in Table III.2 The first row of the table shows that the sample size was about 4000 individuals in each group.
A first interesting result is that a considerably higher percentage accepted the offer of a bonus premium than the offer of a wage voucher to give to an employer: 84 per cent versus 65 per cent. Seen relatively, the difference between the two experimental groups became even greater when one could state that 14 per cent of those who were offered the possibility of a bonus also received one, while the corresponding figure for the group with wage vouchers was 3 per cent. Two factors contributed to making these figures so low. In the first place, many could not manage to find a job within 11 weeks and thus were not entitled to the premiums. Secondly, it turned out that surprisingly many of those who were entitled to receive premiums did not take them out:
Woodbury and Spiegelman report that only 54 per cent of the unemployed who qualified for a bonus payment took it, and only 12 per cent of the companies entitled to money for the voucher they were supposed to get from the new employee asked for the money. The latter is rather strange and might be explained by lack of familiarity with the administrative routines for
- 2 9 -
Table in.2 Some results from the experiment in Illinois
Sample size (number entitled to support for the experimental groups) Percentage who accepted the offer to participate Percentage who
received the bonus
Paid-out unemploy ment compensation, dollars
Number of unemploy
ment weeks
Percentage whose unemployment compensation ceased within 11 weeks
Quotient between saved compensation and payed-out bonus
Control group
3.952
2.558
18.3
0.353
Source: Woodbury and Spiegelman (1987).
Exp. group
offered bonus
4.186
0.84
0.14
2.329***
17 0***
0.408***
2.34a
Exp. group
offered voucher
3.963
0.65
0.03
2.446**
17.7**
0.384**
4.29^
a
b
***
* *
Significantly different from 1 Not significantly different from 1
The hypothesis that the difference between the experiment and the control group is zero can be rejected at the 1 per cent significance level.
The corresponding hypothesis can be refuted at the 5 per cent significance
level
payments. One may fear that this was due to the fact that the experiment
was of short duration and that the outcome would have been different if it
had gone on for a longer time.
The next three lines show in three different ways how the periods of unemployment have been affected by the experiment. Note that the figures for the experimental groups are the averages for those who were offered the possibility of making use of a bonus or voucher. It appears that both the experimental groups received lower unemployment compensation, had on the
average shorter periods of unemployment and to a larger extent left
unemployment (ceased to be a compensation recipient) within 11 weeks than the control group. The effects were somewhat stronger for the group whichwas offered a bonus.
Later, more in depth, analysis of the data (Meyer (1988)), has convincingly shown that the probability of leaving unemployment (the "hazard") was
higher for the experimental groups than for the control group during the first
11 weeks. On the other hand no differences could be found after the point in time when the bonus option had ceised.
Thus, the conclusion is that extra incentives to get a job affects job-seeking behavior. Based on traditional search theory, one can expect that these effects appeared for two reasons: either that the reservation wages were lowered for the purpose of quickly getting a job or that the intensity of job-seeking was heightened, or a combination of both these mechanisms. In order to study this more closely, the incomes for the different groups in their new jobs were also examined by Woodbury and Spiegelman. It appeared that there were no differences among the groups —it is unlikely therefore that the effects arose through a lowering of the reservation wages. On the contrary, the most probable explanation is that the intensity of search increased.
Calculations have also been made of how the costs relate to the benefits for
both the premium types. On the cost side, there is the payment of the premiums, while saved unemployment compensation is entered on the benefit
sOnly the group who received individual bonus was included in the analysis.
- 3 1 -
side. As seen in the last line in the table, the quotient between benefits and costs is clearly larger than 1.0 for both experiments. Even though the quotient is larger for the voucher group, it is not significantly different from one. These calculations are based on only real payments of premiums being entered on the cost side, however. If instead one bases them on the assumption that everyone who was eligible for a premium would have taken advantage of this possibility, which might very well have been the case if the administrative routines had been simplified, then the bonus experiment alone would have made the beneHt cost quotient higher than one.
One conclusion from this experiment is that search behavior is affected by economic incentives. This is not to say however that the search theory per se has been tested. To do that would require that the explicit behavioral relations resulting from the theory are the objects of testing.
Another conclusion is that the effect of offering a bonus directly to the unemployed person is greater than the effect of a wage voucher paid to an employer. The relatively low interest in the wage voucher implies that many perhaps do not believe that these vouchers can help them in getting a job.
The net effect for the group who was offered a voucher is positive, however, even though not strongly statistically significantly different from zero. This means that this study yielded different results than the experiment in Dayton, Ohio. One explanation for this difference can be the 11—week limit, which creates even greater incentive to quickly find a job. Another explanation can be that the two experiments are aimed at different groups:
welfare recipients in Dayton and unemployment compensation recipients in Illinois. In the US, there is a much greater stigma attached to being on welfare than getting unemployment compensation. A wage voucher which reveals to the potential employer that the job seeker is on welfare can therefore have negative effects, while a voucher which reveals that the job applicant has unemployment insurance can have positive effects. Entitlement to unemployment compensation is in the USA, as in most other countries, an indicator of previous work experience.
It should also be stressed that the effects which were presented in the table represent the effects of offering a voucher. The real effects which arise are.
however, concentrated to those who have accepted the offer to make use of this possibility in job seeking. As was seen in the table, 16 per cent of those who were given the possibility of getting a bonus, and 35 per cent of those
offered a voucher declined the offer.
In general it holds that the difference in results between an experimental group who has been offered a benefit and a control group who have not been offered this benefit is the following: (percentage who accepted the benefit) times (the effect for those who make use of the benefit). This means that offering a bonus to a group of unemployed persons affects the periods of unemployment by 1.3 weeks (18.3-17.0), while the effect for those who actually used the benefit is 1.55 weeks (1.3 divided by .84). In the same way, the effect of offering a voucher is .6 weeks (18.3-17.7), while for those who actually used the voucher it is .92 weeks (.6 divided by .65).
Methodological limitations
A surprising result was that so few job seekers and companies who were eligible for financial benefits claimed them. It might be because the administrative routines for handling these vouchers had not had sufficient time to develop properly in the relatively short time that the experiment was conducted. This means in particular that the cost benefit calculations for the program must be interpreted with great caution: the costs were considerably reduced by the fact that many did not claim the reimbursement to which they were entitled.
A reservation must also be made for possible indirect effects. If the search intensity increases noticeably among the job seekers, the wage structure in particular, and perhaps even the wage level, can be affected. It is however probable that such effects arise after a certain length of time and were not in evidence during the period the experiment was being conducted. In economic terminology it is probable that the experiment primarily illuminated partial-equilibrium effects rather than general-equilibrium effects. It is of great interest however to highlight even such more limited effects. Since these partial effects were positive, this can motivate studying bonus payments on a larger scale in a future test and by doing so attempt to shed light on possible
- 3 3 -
indirect effects.
One can also hypothesize that there are effects on the possibilities for the control group to get a job, so called displacement effects. An experiment like this cannot capture such effects. In one of three similar experiments which are planned in the USA a comparison with other regions is planned in order to try to capture such effects. Taking the problems of such comparisons into account (see section V), it can be difficult to find indirect effects with any
precision.
1X1.5 The Employment Opportunity Pilot Project (EOPP)io
This project is not a pure experiment but is nevertheless of considerable interest. In particular, the original plans for the project are instructive, even if altered political prerequisites caused them to be abandoned.
The initial aim was to test a suggestion from the Carter administration based on the idea of offering welfare recipients a guaranteed job. The political idea behind the suggestion was that the people on welfare who could work should do so and those who could not find a job should be offered either a subsidized job —first of all in the public sector - or tr£uning. The suggestion had two phases. First the participants should go through a period of intensive job search. This period contained also work testing and ambitious training in how to apply for a job. The training contained various elements such as practice in contracting employers, being interviewed, filling in application forms and exercises aimed at increasing motivation and raising the level of self-confidence. This part of the program could also be followed in the so-called job-hunt clubs. If the search activity did not yield results in the form of a regular job after five to eight weeks, the second phase of the program would begin. At this time a job or vocational training would be offered. After a year, however, this would cease and the participant would once again go through a period of active job search.
'®This presentation is based primarily on Mathematica Policy Research (1983) and secondarily on Burtless and Haveman (1984).
Enlarging a welfare system with a program of this type is naturally a considerable task, with many potential pitfalls and problems. The aim of the EOPP was to test the program in 14 locations and thereby study how it works before instituting it nationwide. The questions which were actualized by the suggestion were among others the following:
— How well does the first phase of the program with active search work?
Which types of search work best?
— How many will get jobs during the first part and how large will the need for subsidized work places and training places be?
— Will it be possible to find the required number of work and training places?
— Will the jobs be productive or just somewhere to be?
— Will the program improve the participants' labor market situation in the long run?
— What demands will be made on other social services such as child care, transportation needs or counselling?
— How will the rest of the labor market be affected? Will the chances for
other low-income workers of getting a job decrease or be facilitated by the jobs which are created for the program participants?
From a methodological point of view, this project is interesting because a research plan was developed to shed light on these questions. Among other things, a selection of ten comparison sites where the program was not implemented was included in this plan. By studying the experimental and comparison sites, various indirect effects could be brought to light. A comparison of the unemployment among low—income workers between the two groups of sitra should result in opinions concerning whether indirect effects had appeared and which direction they had taken.
- 3 5 -
Unfortunately these plans never reached maturity because the political interest supporting the sugg^tion tapered off. Already during the Carter administration, the rules for admission to the program changed several times, which made the evaluation more difficult. The big change came however when the Reagan administration came into power in 1981. The interest in public service employment, which had been the central instrument for maintaining the job guarantee, diminished. On the other hand, the interest in evaluation of the program's first phase remained, which is why this part of the evaluation project was carried out. The problem was, however, - according to Burtless and Haveman (1984) —that the research plan was not as good when it came to evaluating this phase, while its major merits were in evaluating a program with elements of job guarantees. The results presented are therefore not based on an experimental plan, but have been produced with non-experimental methods based in certain aspects on insufficient data.
Results
The program raised employment 10—12 per cent for unmarried women, who constituted the largest group in the program. On the contrary, a simultaneous decrease in welfare dependency could not be detected. Positive employment effects were received even for other groups of low—income workers who had not participated directly.
A social cost benefit calculation showed that the program most likely implied a short—term subsidy from the tax payers to the participants. If, on the other hand, the positive employment effects had been retained for a longer time, the result could have been positive even for the tax payers.
ni.6 The JTPA experiment in the USA "
In 1982 the American CETA program, whose results were presented above, was replaced by the Job Training Partnership Act (JTPA). Since then the
"This presentation is based on Stromsdorfer (undated), Bangsor, Borus, Bloom and Orr (undated). Bloom, Boms, Orr (1987) and Barnow (1988).
