1935 and then the purpose was to make it possible for single women to take care of their children at full time (see Burtless (1988) and Gueron (1978)).
During the 1960's the attitudes to the program were changed, partly because of new views of women's role in society but also because many critics of the program argued that it created unnecessary welfare dependence. In 1967, therefore, the WIN-program (WIN=Work Incentive) was introduced. It
3 9
-included new rules for the benefits so that the Incentives to take on a job increased and measures for job counselling and training for benefit recipients
with older children.
The "work approach" was strengthened further in 1981 when new legislation made it possible for the states to experiment with harder requirements to accept job offers and enter training programs to receive welfare benefits.
Many states used this opportunity and in particular the harder requirements to search for and accept job offers were implemented as classical experiments at several places in the USA. The research company MDRC (Manpower Demonstration Research Company) has managed the experiments. Large-scale experiments were conducted in eight states during the period 1982-87 and smaller experiments were run at three more places. The experiments have been called the workfare experiments, where workfare stands for a program which requires work (or job search) for welfare recipients.
The design of the workfare programs varied a lot between the states and therefore the experiments are not comparable (see Gueron (1987)). To illustrate the methodological designs we choose to present the study done in San Diego, California (see Goldman et al. (1986)). In San Diego two types of mandatory programs were tested. One of them contained only mandatory job search in structured form during three weeks. The first week was spent providing general information about the labor market and about how to search for jobs. The other two weeks were spent actively searching for jobs in
"job-hunting clubs". The second program contauned an additional
requirement, namely to work for up to 13 weeks at a public or private non profit organization after the first three-week period of job search. Only the welfare benefit was received as salary and the working time was decided as the welfare benefit divided by the legal minimum wage.The methodological design could be made very simple. Those who applied for welfare benefits during the period October 1982 to August 1983 were randomly assigned to two experimental groups and one control group; one of the experimental groups had to participate in the program with only mandatory job search and the other had to participate in the program with both job search and work.
Table III.3 Some results from the workfjare-experimeat in San Di^o
Job search/Work exo Job search
Exper, Control Exper. Control
Diff.
group group Diff. group group
Percentage employed during
1st quarter 35.5 33.1 +2.4 32.9 33.1 -0.2
2nd " 35.6 28.7 +6.9*** 37.2 28.7 +8.5***
3rd " 40.2 32.3 +7 8*** 36.9 32.3 +4.6**
4th " 42.4 36.9
+5!5***
38.4 36.9 +1.65th " 42.9 37.5 +5.4*** 37.9 37.5 +0.4
6th " 41.9 38.1 +3.8* 37.4 38.1 -0.7
Income
during
1st quarter 359 337 +22 368 337 +31
2nd " 510 369 +141*** 487 369 +118***
3rd " 701 538 +163*** 656 538 +118**
4th " 810 693 +117** 669 693 -24
5th " 848 729 +119** 742 729 +13
6th " 933 773 +160*** 799 773 +26
Percentage with AFDC during
1st quarter 78.3 80.3 -2.0 79.5 80.3 -0.7
2nd " 64.2 67.6 -3.4* 66.2 67.6 -1.4
3rd " 51.8 56.2 ^.5** 52.2 56.2 -4.0*
4th " 45.8 47.9 -2.0 45.5 47.9 -2.4
5th " 39.5 41.1 -1.7 42.3 41.1 +1.2
6th " 35.0 36.2 -1.2 36.2 36.2 +0.0
Source: Goldman et al.(1986).
* *
* * *
= significantly different from zero at 10 per cent level
= significantly different from zero at 5 per cent level
= significantly different from zero at 1 per cent level
41
-Some of the results are presented in Table III.3. It appears from the table that the experimental group with both mandatory job search and work experience had better outcome on the labor market than the control group.
There are improvements both in employment, income and benefit collection.
On the other hand no marked improvements can be found for the experimental group which only participated in mandatory job search.
A common result in all the Workfare experiments has been that requirements to work have increased income and led to a small but significant reduction of welfare dependence. However, in general no big improvements in the economic standard of the participants have taken place. The concluding judgments of the experiments by Burtless (1988) and Gueron (1987) are that the measures have been cost-effective without having solved the basic poverty problems which motivate welfare payments. They both emphasize that the experiments have been very useful in illuminating the effects of these types of
programs.
in.S Intensified job search assistance service Eskilstuna, Sweden 12
The main aim of this project —the so-called Eskilstuna study — was to measure, in both quantitative and qualitative terms, the contribution that an extended job assistance service can make in solving labor market problems for the long—term unemployed. The study population, comprising up to 400 persons, consisted of the unemployed job seekers who were registered at the employment agency's district office in Eskilstuna, Sweden, for three months or more. The office in Eskilstuna was given a personnel reinforcement for the period of the experiment, March 10 —June 6, 1975. The experimental group used the agency's services on the average 7.5 hours during the experiment period, compared with 1.5 hours on the average for the control group. The latter group received normally-dimensioned service, which is why the study measures the effects of increased service in particular.
i2This presentation is based on Delander (1978).
As the basis for the work in the experimental group, a program of action was worked out with a strong emphasis on increased individual information and counselling to both job applicants and companies. The program was described
in Delander (1978:192) as follows (translated here from the Swedish):
1 The companies are prompted to change their demands for manpower through intensified information about the capacity and abilities of the job applicants.
a Receiving recruitment personnel at the employment agency where job applicants are introduced.
b Information about individual job applicants given to the agency's representatives participating in cooperation groups in companies seeking personnel.
2 Mobile job search assistance work aimed at influencing the preferences of the job seekers and employers, implying increased information through:
a Intensive direct contact between applicant, agency representative and company, which can mean i.a. that the representative introduces the applicant to the company.
b Group activity for the unemployed aimed at i.a. influencing the professional preferences of the applicant. Study visits to the work
place.
3 Effect of demand with the help of wage—cost-connected subsidi^.
a Acquisition of half-sheltered work, allowances for work assistants, technical aids and special arrangements at the work place and for
vehicles.
b Trial employment, individual training in the company for older and handicapped, purchase of available training power.
4 3
-c Allowance to employers who hire and train men and women in jobs which are dominated by employees of the opposite sex.
The randomly selected experimental and control groups include 216 and 194 persons, respectively. Information about them was gathered at the beginning of the experiment, at the end, and about 9 months later with the help of a mail questionnaire, where the non-response rate was only 10%.
Results
On the whole, this experiment produced considerable positive effects. An overview of the most important results is pr^ented in Table III.4. The first four lines give different quantitative measures of the effects of the experiment, such as employment and unemployment. All the measures indicate clear differences to the advantage of the experimental group, that is, positive effects. It is interesting to note that from lines 5—7, the quantitative improvements do not influence the quality in work placements in the form of wages, job security and work environment. On the contrary, improvements regarding both the quickness and the quality in the placements occurred. For the most, these differences between groups were reported to be significantly different from zero, that is, it is unlikely that mere chance caused the difference between groups.
Delander also carried out some social cost benefit calculations for the
profitability of the efforts. Even these showed a strongly positive result. The costs were less than the production value which was created, and also lower than that part of the production value which accrued "the rest of the society", that is, those who could not participate in the intensified job
assistance activities.
Methodological limitations
A first objection to these very strong positive effects is that the experimental group can have been favored at the expense of the control group. This is a typical case of an experiment where the so-called queue bias can arise.
Delander discusses this possibility and points out that the pressure of demand
Table in.4 Some central results from the Eskilstuna study
1 Percentage with work on the open ma^et at the end of the experiment period
2 Percentage with work on the open ma^et after 9 months 3 Weeks of unemployment from
the beginning of the experiment to follow-up
9 months after the end
4 Average number employment
months for those who were
employed at the follow—up 5 Average monthly wage at
the follow-up for those employed then
6 Percentage with permanent jobs at the follow-up among those with employment at that time 7 Percentage with negative
judgment about the quality
of the work
Source: Delander (1978).
Experimental
group
0.48
0.50
11 weeks
Control group
0.34
0.38
18 weeks
9.0 months 7.6 months
3 588 SEK 3 386 SEK
0.92 0.68
0.12 0.27
4 5
-for labor was strong at the time of the experiment, which is why competition for the available jobs was perhaps limited. At the least, one can assume that the queue bias would have been a more serious problem in a period of deep economic depression.
One way of examining this could have been to study time series for
"probability of leaving a 3-month—long period of unemployment" for the Eskilstuna district. If a shift can be noted for this variable during the experiment period which corresponds to the effect which the comparison between the groups yielded, it is not probable that the effects were achieved at the expense of the control group. Whether the statistics produced by the employment offices at this point in time were of sufficient quality for such an analysis is not clear. In general with the help of such statistics one can supplement the analysis which a pure experiment offers.
Taken as a whole, one can not exclude the possibility that indirect effects appeared in this study, and therefore the size of the effects must be interpreted with caution. Notwithstanding is the fact that acquiring knowledge about the direction and character of the effects (in particular the combination of quantitative and qualitative effects) is of great value. For example, the type of negative stigma effects which arose during the experiment with wage vouchers in Dayton, Ohio, USA, can be completely
eliminated.
Another question is how important it was that some of the wage subsidies were included in the action program which was the starting point for the work with the experimental group. In case they were larger than in the control group, then it is the effect of both the job search assistance activities and the wage subsidies which are included in the analysis. This does not affect in any way the value of the experiment for showing how labor market policy works, only for how it should be interpreted. Delander reports however that both groups used wage subsidies to more or less the same extent.
Yet another question is if the experimental group was representative for the group who would have received more job assistance service if a permanent
reinforcement had been added at the agency. In one respect this can be questioned. If such a reinforcement had been provided to support the long-term unemployed, the resources would probably have been divided among the more than 400 who comprised the total population according to the needs of the applicants for extra service. The experiment meant however that the resources must be limited to 200 persons. It is possible therefore that some in great need of service got none, while some "low-need" persons in the experimental group were given the service instead. In general this ought to lead to an underestimate of the effects, since the experiment meant that the ability to allocate their resources to those with the greatest need was taken from the staff. The larger the spread for need (or the effects of job assistance service, to be more precise) and the better the staffs ability to observe this spread, the greater the potential for underestimation.
Another question one can ask oneself is if the Hawthorne effect could have occurred, which in this case would have primarily concerned the staff and direction of the district office. If they were especially enthusiastic about the experiment, or saw a chance to get extra resources if the results were good, the results can be misleading. One more prerequisite for the results' being completely reliable is that the staff were equal quality-wise in both groups.
In the experiment in Dayton, Ohio, the problem was eliminated by rotating the staff among the different groups. The report does not take up these problems, so one may assume that they were not considered serious in reality.
The ethical objection to an experiment of this type naturally are not great.
To be sure, the control group was denied service which the experimental group got, but in the first place, this occurred by drawing lots (which is usually considered fair), in the second place the experiment lasted for only a limited time, and in the third place, the experience acquired through the experiment was of great ethical value.
4 7
-IV Evaluation of evaluations with the help of experiments
One unanticipated and unplanned use of the National Supported Work Demonstration is that its data made it possible to compare the results between experimental and non-experimental studies, thereby enabling a type of "evaluation of evaluations" to be made. The approach was as follows:
instead of the randomly selected control group from the experiment, a comparison group of the type that one would have used in a non-experimental study was chosen. The data on these individuals were merged with the data on the experimental group. The regression methods and matching techniques which are normally employed in non-experimental studies are then applied to this analysis material. A method which yields the same results as a pure experiment is considered reliable, while the opposite is naturally the case for methods which strongly deviate from the results of the experiment. This is of course based on the assumption that an experiment which is designed like the National Supported Work Demonstration gives a completely correct estimate of the program's effect. As was seen in the presentation in section III.2, there are good reasons for having confidence in this experiment.
This approach to evaluations of non-experimental methods has been employed by Fraker and Maynard (1987) and Lalonde (1984 and 1986).
Fraker and Maynard concentrated on two sub-groups from the project, namely women with AFDC and youths. The comparison groups were chosen from the Current Population Survey, the American labor force survey. The basis for the selection of the group was the same as the fundamental criteria for admission to the program: youths should be between 17 and 20 years of age, and women with AFDC should not have children under 6 years of age.
They started by studying the results of a regression model where the outcome in incomes (1977, 1978 and 1979) were explained not just by participation in the program, but also by demographic characteristics, previous experience, experience of unemployment, previous incomes, and changes in income (prior to the program). In Table IV.l, the effects which are estimated with this model are presented. The most remarkable is the big discrepancy between the effects received from the control group and those from the comparison group for the youths. In fact, the results deviate so greatly that the
non-experimental approach must be considered completely unreliable.
On the other hand, the results from the control group and the comparison group tally relatively well for women with AFDC. The discrepancy is greatest for the third year, when the effect received with the comparison group is about 25% lower (calculated in dollars) than the effect that the experiment generated. Even if the point estimates are more or less in agreement, one can also see from the standard error that precision is relatively low. This source of uncertainty in the estimated effects is not discussed however by the authors.
Table IV.l Estimated effects (in dollars) of the National Supported Work Demonstration with the help of a randomly selected control group and a comparison group taken from the Current Population Survey
Youths Women with AFDC
Year Control group Comparison g. Control group Comparison g.
1977 313 -668 1423 1560
(134) (310) (162) (400)
1978 -28 -1 191 505 537
(135) (373) (137) (335)
1979 -18 -1 179 351 257
(166) (375) (174) (465)
Note: Standard errors in parentheses.
Source: Fraker and Maynard (1987).
4 9
-Fraker and Maynard also study some alternative regr^sion models^^ and comparison groups. They found that the results were sensitive to the choice of comparison group, particularly for youths. Some sensitivity for the choice of regression model could also be noted.
Thus the main conclusion from this study is that regression models which use longitudinal data can replicate the results from classical experiments relatively well for women, but badly for youths. In one way this is hardly surprising, since the incomes prior to the program can be expected to be less valuable as a control variable for youths than for adult women. The purpose of using previous incomes as control variables is that they constitute a measure of the individual's position on the labor market; since the youths have hardly had time to establish themselves on the labor market, this information is of little value as a control variable for this group.
The second study —presented in Lalonde (1984 and 1986) —was based on the group of women with AFDC and on men who were part of some of the other groups. Lalonde constructed the comparison groups by randomly sampling women with AFDC and men from the 1976 Current Population Survey. The criteria for being included in the comparison groups were to have belonged to the labor force in March 1976 and to have had an income below USD 20,000
in 1975.
When Lalonde estimates the effect of program participation with the help of regression models and the above mentioned comparison group, he finds once again that it is easiest to reproduce the results from the pure experiment for the group of women with AFDC. Since the group of men had a rather low average age, 24 as opposed to 34 years for women with AFDC, the reason is probably the same as above. Earlier incomes which are used as control variables are most likely more informative for adults than for the young.
Lalonde also compares how well different regression models are able to reproduce the results of the experiment. The first model employed cross-sectional data only: incomes after the program were explained by the program participation as well as age, age squared, race and education. The
'^So—called first difference models (with and without changes in the explanatory variables).
second used longitudinal data by explaining the change in incomes (from a point in time before the program to a point in time after it) as a function of
second used longitudinal data by explaining the change in incomes (from a point in time before the program to a point in time after it) as a function of