Study-Related Groups - Bivariate Statistics

Case II - Effect Does Exist

3.5 Bivariate Statistics

3.5.1 Study-Related Groups

Although to penetrate into the intimate mysteries of nature and thence to learn the true causes of phenomena is not allowed to us, nevertheless it can happen that a certain fictive hypothesis may suffice for explaining many phenomena.

Leonhard Euler, Introductio in analysin infinitorum, 1748

In the following, we will present the dependency within certain study-related groups. This means that the grouping variable is constant within each study. We inspect the relationship be-tween several interesting variables and the (normalized) t-values and comment on it.

As depicted intable 3.26, compared to U.S. authors, studies from authors of the most frequent and largest European countries (the United Kingdom and Germany) yield significantly better re-sults (in favor of the deterrence hypothesis). Their medians are−1.85 and−1.83, which is almost

38The user tr recorded all economic studies, the user aw most of the criminological and sociological studies.

Table 3.25: Significant correlations with the (normalized) t-values

Variable coef. variable coef.

Study: size of first realized sample −0.235^?? Study: publication, year −0.056

Study: size of second realized sample −0.119^?? Study: author, Isaac Ehrlich −0.056

Study: user, tr −0.113^?? Study: complete sample −0.055

Study: publication, economics −0.110^?? Estimate: exogenous, survey, other −0.055

Study: error and plausibility checks −0.103^?? Study: journal, Review of Economics and Statistics −0.053 Estimate: sub-sample of youths −0.103^?? Estimate: deterrence is focus-variable −0.053

Study: institute, economics −0.099^?? Estimate: exogenous, binary category −0.050

Study: journal, Economic Inquiry −0.097^?? Study: cross section +0.050

Estimate: number of observations −0.093^?? Study: sample individuals, first population, pupils +0.051 Estimate: covariate, Fixed effects (spatial) −0.092^?? Study: institute, criminology +0.051

Study: author, economics −0.092^?? Estimate: exogenous, not in logs +0.052

Study: economic, rational choice theory −0.091^?? Estimate: exogenous, death penalty, existence of death penalty

+0.052

Study: main location>500000 inhabitants −0.091^?? Study: journal, Criminology +0.053 Study: rate of return of second sample −0.088^?? Estimate: deterrence is covariate +0.053

Study: experiment (laboratory) −0.085^?? Study: author, criminology +0.055

Estimate: exogenous, experiment, experimental varia-tion of probability of detecvaria-tion

−0.081^?? Study: user, aw +0.055

Study: author, psychology −0.080^?? Estimate: exogenous, death penalty, execution rate +0.055 Study: sample individuals, second population,

miscella-neous

−0.076^?? Study: author, William C. Bailey +0.056

Estimate: exogenous, experiment, yes −0.076^?? Estimate: multivariate method, path analysis +0.057 Estimate: covariate, marital status −0.076^?? Estimate: covariate, urbanity +0.061^? Estimate: exogenous, experiment, relates to the present −0.075^?? Study: author, sociology +0.063^? Estimate: exogenous, crime data, arrest rate −0.074^?? Estimate: bivariate method, correlation +0.065^? Estimate: covariate, personal characteristics −0.073^?? Estimate: covariate, previous convictions +0.066^? Study: institute, psychology −0.072^? Estimate: exogenous, crime data, police expenditures +0.067^? Estimate: exogenous, other transformation −0.068^? Estimate: test of significance +0.067^? Study: sample unit, second population, individuals −0.067^? Study: institute, miscellaneous +0.069^?

Study: experimental −0.066^? Study: sample unit, first population, states +0.071^?

Estimate: endogenous, other −0.066^? Study: journal, Criminal Justice +0.072^?

Study: sample unit, first population, individuals −0.066^? Estimate: covariate, poverty, welfare +0.073^??

Estimate: exogenous, in logs −0.065^? Estimate: study type, death penalty +0.079^??

Study: publication, working paper, report −0.064^? Study: not experimental +0.082^??

Study: tests of significance −0.061^? Estimate: weighted model +0.088^??

Study: sample base, second population, complete coun-try

−0.060^? Study: traditional theory +0.111^??

Estimate: exogenous, crime data, conviction rate −0.059^? Estimate: exogenous, crime data, police strength +0.126^??

Estimate: multivariate method, logit, probit −0.058 Study: publication, criminology +0.130^??

Study: first population, United Kingdom −0.057^?

A correlation coefficient is listed if its absolute value is larger than 0.05, is significant at the 0.01% level (two-sided test) and varies in at least 1% of the data. ^?marks p-values which are smaller than 5·10⁻⁶and ^??those below 5·10⁻⁹.

40% smaller than those of U.S. authors. Contrarily, Canadian authors find deterrent effects in much fewer cases (the percentage³⁹ is reduced by one third, compared with those authors from the UK and Germany). Estimates from Australian authors do not significantly differ from those of U.S. authors. The statistics of the authors from less frequent countries (“other”) are also in-teresting: the mean and median estimate is much more negative than those from U.S. authors, while the percentage is much lower. This could mean that the results from those authors are more concentrated in the negative “no man’s land”, which is rather uncommon in our meta-data base (refer tosection 3.4about publication bias).

Table 3.26: Differences by the authors’ nationality

Nation mean median % #e #s

Finland −3.03 −2.92 80.00 50 5

Israel −1.64 −2.15 53.60 85 9

UK −1.87 −1.85 48.76 275 28

Germany −1.86 −1.83 42.15 208 22

Netherlands −1.11 −1.46 43.58 81 8

Other −1.81 −1.44 34.91 239 24

Overall mean −1.40 −1.37 41.66 6530 663

USA −1.38 −1.29 41.90 5143 522

Australia −1.30 −1.01 44.87 132 13

Switzerland −0.80 −0.99 38.81 66 7

Canada −0.86 −0.89 31.22 336 34

Sweden 0.15 −0.15 7.87 61 6

Mean and median correspond to the (normalized) t-values of the particular group.%is the percentage of estimates which are consistent with the deterrence hypothesis and significant at a 5% level in a two sided test. #eis the weighted number of all valid estimates. #sis the number of studies the estimates are based on. Italicentries are not significant in the ANOVA at a 5% level, but are included, like the overall mean, to show otherwise interesting or some selected groups. The underlined entry is the reference category.

As mentioned before, it is almost common knowledge that economists more readily agree with the deterrence hypothesis than criminologists and sociologists. This view is supported by ta-ble 3.27. Psychology is, undisputata-ble, number one in that list with a median of −2.15 and a percentage of 65.59%. These authors mostly study alcohol related offenses which yield rather negative results. Economics is the second most dominant category in favor of deterrence; the estimates have a median of−1.67 and 43.82% are consistent with the deterrence hypothesis and significant. As expected, sociologists show significantly less results in agreement with deterrence (median−1.01, percentage 38.67%). With a median of just −0.62 and a percentage of 33.31%, criminologists are at the very bottom of that list. Authors from “other” disciplines, which are more or less not related to deterrence research (e.g., mathematics, medicine, etc.), produce esti-mates which are very similar to the overall mean.

39In this context, “percentage” always refers to the percentage of (normalized) t-values which are consistent with the

Table 3.27: Differences by the authors’ discipline

Discipline mean median % #e #s

Psychology −2.58 −2.15 65.59 275 27

Economics −1.73 −1.67 43.82 2807 287

Overall Mean −1.40 −1.37 41.71 6101 617

Other −1.17 −1.33 41.25 767 76

Law −1.17 −1.11 38.53 233 23

Sociology −1.07 −1.01 38.67 1690 170

Criminology −0.93 −0.62 33.31 738 75

As stated in section 3.4 about publication bias, results may be different for various types of publication. Journals make up for the most studies we included (table 3.28). Working papers (including discussion papers, official reports, etc.) is the only category in which the (normalized) t-values are significantly different (the mean of−2.26 is quite smaller than−1.34 from the jour-nals); but this difference is almost nullified when we look at the median or the percentage (−1.37 and 43.43% compared to−1.35 and 41.74%). All in all, there seems to be no major differences between the various types of publication; only that the average (normalized) t-value is more neg-ative (−1.93) for books (the median of 1.57 and the percentage of 46.38% are different, but not significantly).

Table 3.28: Differences by the type of publication

Type mean median % #e #s

Edited volume −1.62 −1.63 43.82 261 28

Book −1.93 −1.57 46.38 98 10

Other −1.19 −1.41 31.89 214 22

Working paper −2.26 −1.37 43.43 322 34

Overall mean −1.40 −1.37 41.66 6530 663

Journal −1.34 −1.35 41.74 5635 569

Most studies are published in five countries, dominated by U.S., as shown intable 3.29.

How-deterrence hypothesis and significant at a 5% level in a two-sided test.

ever, there are almost no differences. Only the Netherlands stick out with a very low percentage (28.99%) and an average (normalized) t-value near zero (−0.46). An interesting observation is that studies in Canadian publications report more negative (normalized) t-values (median−1.67) than the average, while authors from Canada report less negative (−0.89) values.

Table 3.29: Differences by the country of publication

Country mean median % #e #s

UK −1.53 −1.82 47.57 539 54

Canada −1.42 −1.67 39.37 234 23

Overall mean −1.40 −1.37 41.66 6530 663

Germany −1.50 −1.34 36.92 299 31

USA −1.41 −1.29 41.84 5101 518

Netherlands −0.46 −0.96 28.99 166 17

Other −1.59 −0.94 41.16 190 20

We distinguish the discipline of the publisher intable 3.30. Economic (and psychologic) pub-lications are much more supportive of the deterrence theory (−2.15 and −1.73, 67.45% and 45.50%), while sociological (−1.33, 40.64%) are significantly less supportive. Criminological publications appear, after a large gap, at the very bottom of the list (−0.51, 30.85%). It is not as-tounding that there are no major differences totable 3.27because sociologists and criminologists only rarely publish in economic media (as illustrated intable 3.9).

Table 3.30: Differences by the publishers’ discipline

Discipline mean median % #e #s

Psychology −2.12 −2.15 67.45 183 18

Economics −1.86 −1.73 45.50 2348 241

Other −1.54 −1.42 45.24 945 96

Overall mean −1.40 −1.37 41.70 6426 651

Sociology −1.15 −1.33 40.64 1267 128

Law −1.56 −1.20 38.25 346 34

Criminology −0.62 −0.51 30.85 1335 134

Intable 3.26, we have seen the relationship with the countries the authors worked in. Although many authors study data of their own country, there are some differences. Looking attable 3.31, which shows the statistics diversified by the studied countries, some countries remain fairly stable in their position: Finland remains at the top (median−2.65, percentage 83.40%), while the UK, Germany and the USA remain at their positions as well. Australia switches from below the mean almost to the top (from −1.01 to −2.24 and from 44.87% to 64.39%). Sweden remains at the lower part of the table, but has now a more reasonable (higher) percentage (7.87% to 24.65%).

Studies using Canadian data have estimates which are in least agreement with the deterrence hypothesis (−0.52, 28.26%). Although some crimes are studied more often in a country than in another (e.g., drunken driving in scandinavian countries or marijuana consumption in Australia), there is no obvious explanation why results based on Canadian data are in “worst” compliance with the deterrence hypothesis.

Table 3.31: Differences by the studied nation

Nation mean median % #e #s

Finland −2.93 −2.65 83.40 60 6

Australia −1.74 −2.24 64.39 112 11

UK −2.17 −2.02 54.07 327 33

Germany −1.96 −1.83 42.24 177 19

Netherlands −1.79 −1.37 41.98 81 8

Overall mean −1.40 −1.37 41.66 6530 663

USA −1.37 −1.30 41.84 5008 507

Other −1.37 −1.19 31.88 407 43

Switzerland −0.97 −1.03 38.81 66 7

Sweden −1.31 −0.69 24.65 92 9

Canada −0.70 −0.52 28.26 280 28

During the last 40 years much has or may have changed in the literature about deterrence:

data, data quality, cultural backgrounds, estimation technology, attitudes of authors, the audience, offenders, and much more. Figure 3.11, however, reveals that there are no obvious time effects.

Nevertheless, we have partitioned the estimates into five categories⁴⁰. There are indeed some significant differences, as shown in table 3.32, but the differences are neither easily interpreted, nor are these very pronounced. If at all, there is a periodical pattern. However, these results are not robust when redefining the time categories.

40As always in such cases we chose the thresholds in such a way that each category contains an approximately equal number of observations.

Figure 3.11: Temporal development of the (normalized) t-values

Overall mean

−15−10−5051015Normalized t−value, cut at |15|

1950 1960 1970 1980 1990 2000

year

t−value Median band

Based on own meta data base

Temporal development of the results

We make the same comparison with the utilized data. Table 3.33shows that the median values differ as much as in the case of the year of publication; the percentages only partially. Although the entries are all significantly different from the newest data (median−0.92, 36.49%), there is a slight trend in the order of the rows. If we exclude the oldest data, studies using newer data seem to produce less significant results. Additionally, the ordering is somewhat different from that in table 3.32because there is, if at all, a time trend instead of a periodic pattern.

The number of reported estimates certainly depends on the type of publication, because there is much more room to present results in books and working papers than in journals. The mean number of published estimates is 22 (median 8); the largest number is 764 (the smallest zero).

The corresponding ANOVA can be found in table 3.34. Although not significant there is an obvious descending order with an increasing number of estimates. This may be, at least partially, explainable by technical reasons: presenting more results is commonly done for robustness checks (often by economists), which come along with more “contaminated⁴¹” estimates. Another reason is that simple correlation coefficients, which are associated with insignificant values (table 3.47), appear often in large numbers in a study.

41When evaluating numerous specifications a certain percentage may suffer from a misspecification bias.

Table 3.32: Differences by the year of publication

Years mean median % #e #s

1979-1986 −1.42 −1.68 46.26 1225 125

1987-1994 −1.36 −1.53 43.05 1428 147

2001-2006 −1.90 −1.46 44.97 1412 141

Overall mean −1.40 −1.37 41.66 6530 663

1952-1978 −1.12 −1.18 39.25 1237 125

1995-2000 −1.15 −1.06 34.06 1229 125

Table 3.33: Differences by the year of utilized data

Years mean median % #e #s

1966-1974 −1.11 −1.61 43.71 962 97

1975-1982 −1.17 −1.31 42.93 1023 103

1983-1988 −2.21 −1.20 41.73 824 85

Overall mean −1.33 −1.19 39.25 4862 494

1875-1965 −1.05 −0.94 32.07 998 103

1989-2004 −1.25 −0.92 36.49 1056 106

It may be reasoned that the design of a study may lead to different results. It is obvious that experiments come along with very negative (normalized) t-values (−2.1 to −2.45, 58.57% to 65%), astable 3.35shows. The effect of the other designs are not that obvious. Surveys appear at the upper, middle and lower part of the table, depending on their design. Estimates based on reported crimes are below the mean. However, most of the non-experiments are not significantly different from studies based on time series (the reference category,−1.34, 41.69%).

Section 3.3shows that 27.6% of all studies come from the top 22 authors (2.6% of all involved authors). It is reasonable to assume that individual preferences of the authors may lead to different estimates, depending on their personal attitude and other reasons. We stress thattable 3.36does not show any (clear) evidence for a publication or author bias; authors may prefer different meth-ods, offenses, countries and other things which may lead to more or less significant estimates.

The most striking result is that, among these 22 authors, the percentage of theory-consistent and

Table 3.34: Differences by the number of reported estimates

Number mean median % #e #s

0-2 −1.58 −2.09 53.96 1349 133

Overall mean −1.46 −1.41 42.82 5320 540

3-5 −1.25 −1.41 42.11 791 79

6-11 −1.29 −1.33 39.69 967 98

12-25 −1.60 −1.10 40.17 1021 104

26-764 −1.49 −1.00 35.49 1192 126

Table 3.35: Differences by the design of a study

Design mean median % #e #s

Experiment (by institution) −2.55 −2.45 63.33 102 10

Experiment (laboratory) −2.63 −2.36 58.57 285 28

Experiment (by researcher) −1.03 −2.10 65.00 102 10

Survey (once) −1.55 −1.60 45.63 1161 119

Experiment (natural) −1.26 −1.60 44.90 197 20

Overall mean −1.41 −1.37 41.69 6479 658

Time series −1.39 −1.34 39.06 2235 224

Survey (panel) −1.50 −1.32 44.52 367 38

Panel data −1.57 −1.13 38.68 1000 101

Cross section −1.12 −1.04 37.00 1517 156

Survey (multiple) −1.46 −0.86 36.15 205 22

significant results is almost doubled in the top five rows compared to the bottom five. While the upper position of Isaac Ehrlich (−3.34, 73.65%) and William C. Bailey (−0.26, 13.92%) in the last row is not surprising, it is rather curious to find Steven D. Levitt (−0.83, 32%) among the bot-tom five. As stated before, there are only two authors who did not live in the USA at the time of writing and both appear above the mean (−1.37, 41.66%); Matti Vir´en has the highest percentage (80%), while Horst Entorf⁴² is slightly above the mean (42.96%).

42Although his entry is just below the mean and the reference group, the corresponding mean and percentage indicates that he has to be associated, with the “upper” part.

Table 3.36: Differences by prominent authors

Author mean median % #e #s

Simon Hakim −2.91 −3.47 63.82 67 7

Isaac Ehrlich −3.13 −3.34 73.65 64 7

Matti Vir´en −3.03 −2.92 80.00 50 5

Harold G. Grasmick −2.14 −2.58 72.61 101 10

Dale O. Cloninger −1.77 −2.23 63.79 97 11

Laurence H. Ross −2.39 −2.10 67.74 63 7

Greg Pogarsky −1.50 −2.03 53.24 61 6

Daniel S. Nagin −0.73 −1.79 45.19 87 9

Theodore G. Chiricos, Gordon P. Waldo −1.55 −1.61 49.51 81 8

David W. Rasmussen −1.64 −1.45 28.13 81 8

Bruce L. Benson −1.61 −1.45 29.94 92 9

Charles R. Tittle −0.90 −1.38 47.71 61 6

Overall mean −1.40 −1.37 41.66 6530 663

Raymond Paternoster −1.55 −1.33 37.01 156 16

Other −1.38 −1.29 40.69 5058 513

Horst Entorf −2.51 −1.20 42.96 55 6

Ann D. Witte −1.32 −1.20 38.21 66 7

Maynard L. Erickson −1.22 −0.87 34.29 41 4

Jack P. Gibbs −1.13 −0.85 31.19 61 6

Steven D. Levitt −0.61 −0.83 32.00 112 11

Thomas B. Marvell −1.17 −0.76 39.56 61 6

Alex R. Piquero −0.84 −0.61 31.42 81 8

William C. Bailey −0.35 −0.26 13.92 173 17

Chiricos appear together in all of their studies.

As mentioned before, we have build an index which relates to the subjective quality of each study. It consists of the magnitude and quantity of problems reported by the author and the extent of unreported problems judged by the reader (i.e., the user who recorded the study). We have aggregated this index into three categories: good, medium and poor quality. Again, we emphasize that there is no such thing like a flawless study; not all problems can be coped with by corrective measures and some may lie deep in the available data source. While the estimates of poor studies differ significantly from those of medium quality, the order intable 3.37is, all in all, somewhat inconclusive. Studies of medium quality (−1.25, 39.44%) are less in favor of the deterrence theory than those of poor (median−1.59, percentage 46.41%) and good quality (−1.65, 45.25%).

Although the averaging effect of the sheer size of the category of medium quality may partially explain this, the order remains a bit strange. The first and last rows are practically identical (in

regard to the mean (normalized) t-value) while their median and percentage differ.

Table 3.37: Differences by the quality of a study

Quality mean median % #e #s

Good quality −1.38 −1.65 45.25 1840 187

Poor quality −1.77 −1.59 46.41 538 56

Overall mean −1.40 −1.37 41.66 6530 663

Medium quality −1.37 −1.25 39.44 4152 420

Each study has to rely on some data. This can be public crime data (like the UCR in the USA, the PKS in Germany, public surveys or non-public data like the combination and linkage of various data sources, confidential data, experiments, self conducted surveys etc.Table 3.38shows that there are no significant differences in the estimates when we categorize them according to the implemented data source. Nevertheless, it is worth mentioning that studies using the UCR come along with the smallest (normalized) t-values (−1.07, 36.99%), while those estimates based on non-public data yield “better” values (−1.68, 47.29%).

Table 3.38: Differences by the public data base

Data base mean median % #e #s

PKS −1.93 −1.78 37.32 75 8

None −1.46 −1.68 47.29 1714 175

Overall mean −1.38 −1.33 41.04 5900 600

Other −1.33 −1.31 39.36 2618 264

UCR −1.33 −1.07 36.99 1494 153

As already described insubsection 3.3.3, we have an index of the general opinion of the author at our disposal, aggregated over all crimes.Table 3.39is more or less a verification and plausibility check. As expected, all categories are significantly different from the reference category (partial approval) and all three statistics are in descending order (from −2.66, 64.5% to 0.32, 8.02%).

Even theUndefined category is in the middle (−0.91, 35.41%), although it is not to be mixed up with something like “indifference”; it accumulates all not unambiguously definable opinions:

usually these happen to be studies lacking any usable statements which could reveal the opinion of the author or, which happens quite often, the opinion depends heavily on various conditions.

Table 3.39: Differences by the overall author opinion

Opinion mean median % #e #s

Full approval −3.07 −2.66 64.50 1377 142

Partial approval −1.87 −2.08 54.48 2151 215

Overall mean −1.40 −1.37 41.66 6530 663

Undefined −0.75 −0.91 35.41 832 86

Partial disapproval −0.58 −0.52 24.54 1158 118

Full disapproval 0.37 0.32 08.02 1012 102

Im Dokument Meta Analysis of Crime and Deterrence : A Comprehensive Review of the Literature (Seite 128-139)