Employment Opportunity Pilot Program. This program contained a job guarantee; in order to investigate how many who could be expected to make use of this opportunity, the program must be allowed to operate in its entirety for a reasonable length of time. Therefore it is natural to choose a number of sites where the program is tested fully.
A third plus with macroexperiments can arise if the measure which is the object of study will be handled strictly administratively in a more natural way if it can be applied in its entirety in one place or district within the labor market policy organization. It is possible that the assignment into an experimental and a control group within one district, as fits the requirements of microexperiments, is administratively bothersome. It is difficult to make any general statements about this problem - it must be judged from case to
case.
The main problem with macroexperiments is that it is difficult in practice to get reasonable sample test sizes for the experimental and the control group.
The reason for this is that a city from a purely statistical point of view must be considered as one observation. Even if a place, chosen at random, with e.g.
25,000 inhabitants is the experimental town for a certain measure, it must be considered as one independent observation only. If on the other hand 25,000 individuals are randomly selected for a certain measure, purely statistically they constitute 25,000 independent observations. When one estimates the difference in the outcome between the experimental and the control groups for a certain variable —take employment —the estimate is perhaps in practice based on just a few observations. In this case the precision will be low and it might be difficult to make meaningful statements about possible effects. In practice some difference between the groups is of course always registered. If however one wishes to make statistical inference about whether the registered difference can be a result of chance or a pure effect, small sample will create problems. The problem is worsened in case the "natural" spread in the result variable is big, which is often the case when it is a question of towns or regions.
A related problem with macroexperiments is that it may be hard to really
caxry out a probability sample of towns. One prerequisite is that a population of towns can be defined in a meaningful way. If this is possible, the result will most likely be a number of towns with very varied characteristics and different sizes. Two randomly selected groups from such a population will most probably indicate a considerable "natural" variation in outcome variables such as employment, unemployment and periods of unemplojmient.
A tempting solution could be to choose towns with similar characteristics for both the groups, a sort of matching procedure's. The problem with the procedure is that one must first abandon the principle of simple random sample, which makes it more difficult, and in some cases impossible to make a statistical analysis. It should be noted that the towns which were chosen to be part of the experimental and control groups in the EOPP project were not selected randomly.
Taken as a whole, macroexperiments appear as an elegant way of avoiding the problems with indirect effects, but in practice it can be hard to test the differences in outcome between the experimental and control groups statistically with good precision. In statistical terminology this means that it will be difficult to "reject the zero hypothesis that no effects (i.e. no differences between the groups) exist." Only if the actual effects are very large can it be possible to reveal them with the help of macroexperiments.
For certain types of labor market policy measures macroexperiments appear logically speaking as the only solution, however. This is the case if one wants to study the effects of new work forms at the employment agencies for example. Here towns are the natural unit of analysis and macroexperiments the natural form of analysis.
'9An example of this is the so-called LYZS project conducted by the Swedish National Labor Market Board. Here four counties were chosen —probably not randomly —where new rules for unemployed's follow—up visits were in effect and four "similar" counties which followed the older regulations. To apply statistical method to study the differences in the outcome variables between groups chosen in this way is not easy.
5 7
-VI Suggestions for future evaluation research
The evaluation research concerned with vocational training courses for the unemployed and other individual-directed measures which has dominated within the economic journal literature has been relatively ambitious. The statistical methods have been quite well developed and the links to economic theory have been improved. Further, at least for the American CETA program, initiative was taken for extensive data gathering which gave the researchers access to longitudinal data bases containing information on employment and incomes both before and after the program for participants and non-participants. Despite this it has been difficult, as shown above, to produce really reliable estimates of the effects of the programs. The American studies of the CETA program indicated after all rather uniform results only
for women.
The question is therefore if it is possible to go further and, if so, how. In the research community one can distinguish two basic opinions: one, with Orley Ashenfelter as the leader, advocates classical experiments; the other, with Jim Heckman at the head, pleads for continued work with non-experimental methods. This can raise memories of the 60's and early 70's, when corresponding schools of thought could be distinguished. Has there been no progress at all? Without a doubt considerable progress has been achieved.
Insights into the character of the methodological problems have increased greatly. Regarding experiments, one can cite as examples, as presented above, more than a half a dozen completed projects. These are well documented and have contributed valuable practical knowledge on how experiments should be carried out. In addition, there is a theoretical literature around experiments which has provided valuable insights (see for example Hausman and Wise (1985)). Therefore the chances of succ^sfully conducting experiments from a methodological point of view are better today than 15-20 years ago.
For non-experimental methods as well, there has also been an increase in insights. One can say more precisely now what knowledge of the dynamic structure of the incomes and of the selection process is needed in order to choose among the methods available today.
These two approaches are of course not mutually exclusive — instead, they complement each other, as was seen in section IV. Whenever it is practical and politically possible to conduct experiments, it is therefore a good idea to collect data which could also enable the use of non-experimental methods.
The experiment would provide an opportunity to evaluate the non-experimental methods, as was presented in section IV: This aspect is included in the plans for the JTPA experiment in the USA.
We will conclude now with a more detailed description of how experiments can be designed and how non-experimental methods can be developed for the purpose of providing better knowledge about programs like the Swedish vocational training courses for the unemployed.
VI.1 Experiments
How can one set up experiments with whose help one can quantify the effects of individual-directed measures such as vocational training courses for the unemployed? The problems are of an ethical, organizational and methodological nature. The ethical problems concern whether the experiment denies certain individuals benefits of different types. Organizational problems can arise within the labor market authority which administers the program and which normally must change its way of working. Finally, the methodological problems are those problems which confront the researchers.
The discussion can be facilitated by some fundamental concepts and distinctions. First of all, one can differentiate between ongoing (and established) programs, on the one hand, and new programs, on the other. New programs can also include changes and new elements in ongoing programs.
Depending on the selection procedures for the program, one can distinguish between demand-constrai ned and supplv-constrained programs. In demand-constrained programs, it is the potential participants' interest in the program which decides the participation. This is the case if, for example, the supply of places exceeds the number of demands, so that everyone can participate. The Swedish vocational training courses for the unemployed during the last decade have in general been characterized as demand-constrained. In supply-constrained programs, on the other hand, not
5 9
-everyone who wants to participate and who has applied can get a place.
Instead the authorities who administer the program make a selection among the applicants. This can be the case if the course capacity is not big enough within a training system or if the supply of job counselors at the employment agency is insufficient. In a supply-constrained system, the selection occurs through decisions of both the prospective participants, who indicated their interest in the program and the authorities who choose among the applicants^o.
In a demand-constrained program it is on the contrary only the interest of the participants which controls the selection.
Demand-constrained programs
Consider first an experiment which will shed light on the effect of a demand-constrained program. The natural population and sample frame from which the experiment and control group ought to be randomly selected is a list of the people who applied for the program^i. It is therefore important that the actual application procedure is not affected by the fact that an experiment is to be carried out, but that both the information about the program and the applications are made in the same way as when the program is in normal operation This should result in randomly selected experimental and control groups having on the average the same characteristics; they have gone through the same selection process —that is, they have decided to apply to the program. The experimental group furthermore has the same characteristics as the participants would have had if the program had been operating normally: it is chosen at random from the group who applied and normally would have participated in the program. The difference in outcome - employment and incomes, etc. - between the groups therefore provides a reliable estimate of the effect of the program. More precisely, this difference shows how the program on the average affects the results for the program participants. (Since it is a mlcroexperiment, a reservation must be made however, as indirect effects can bias the results).
An experiment of this type runs up against both ethical and orgeuiizational 2®This choice by the authorities can either be "participate/not participate" or how much service a certain client shall get (for job assistance service).
2iln practice an examination must of course be made to see if they fulfill the