The “how” of field experimental approaches

Based on Harrison and List (2004) we propose a taxonomy of relevant field experiments, which, in their words, “leaves gaps” but is useful for our purpose of contextualizing the two approaches we use in this study. In their article, Harrison and List propose six factors by which the “field context” of an experiment can be characterized: (1) the nature of the subject pool, (2) the nature of the information that the subjects bring to the task, (3) the nature of the commodity, (4) the nature of the task, (5) the nature of the stakes, and (6) the nature of the decision environment.

In Figure 3.1, the most widely used variants of experiments are presented and contrasted based on the underlying data type. For illustrative purposes, conventional laboratory experiments and natural experiments are included in the figure, representing the extreme points on the stratum of experimental approaches, albeit they do not fall in the category of experiments in the field in a narrow sense. Besides that, frequently used econometric approaches named above are also listed to complete the picture.

Starting on the extreme left side of the stratum that indicates the degree of control assumed over the data generating process, we skip the classic laboratory experiment and start the discussion with the field experimental approach that promises the highest degree of control: the artefactual field experiment (AFE). AFE resembles the lab experiment in all dimensions (e.g. abstract framing of the decision task) except that the participants in the experiment are recruited from a non-standard subject pool, which—and this is noteworthy—is the common denominator of all

field experimental approaches. While participants in laboratory experiments are usually recruited from populations of college students (commonly referred to as standard pool), subjects in field experimental approaches are purposefully sampled from a non-standard pool representing the target population in the economy. Two examples for artefactual field experiments are those of Harrison et al. (2002) who carried out experiments in Denmark with a nationally representative sample of subjects to estimate individual discount rates, and Gächter and Herrmann (2011) who conducted public goods experiments with participants in Russia, assuming that the long history of collectivism had an impact on voluntary cooperation.

Framed field experimental approaches (FFE), which we use in this study (Chapter 4), are similar to artefactual field experiments. But in addition to a non-standard subject pool, FFE are characterized by less abstract framing, often including choice tasks mimicking day-to-day decisions as well as more tangibly defined commodities. For example, studying the effectiveness of policies on coca investment, Ibáñez and Martinsson (2008) conducted a framed field experiment in which smallholders in Colombia had to take hypothetical agricultural investment decisions.

The nature of stakes in lab and field experimental studies has been subject to a longstanding debate. Experiments in high-income countries have been frequently criticized, because—due to budget constraints—stakes were relatively low, raising the concern that incentive structures in (laboratory) experiments would not reflect field conditions well. This was confirmed by high-stakes experiments carried out in both low- and high-income countries (Kachelmeier and Shehata, 1992; Holt and Laury, 2002). These findings demonstrate the appeal of carrying out field

experiments in low-income countries, as the researcher can run experiments with non-trivial stakes despite typical budget constraints.

Source: Based on Levitt and List (2009)

Figure 3.1: Typology of experimental approaches in economics

An ideal laboratory setting allows the researcher to minimize noise in the decision task, as the only varying factor is the “stressor” the researcher is interested in (Harrison and List, 2004). In contrast, if experiments are taken to the field there is usually more noise, for example due to relationship between subjects and the decision environment, e.g. the experimental site.

In terms of the decision environment there are major differences between artefactual and framed field experiments on the one hand, and field experiments

(FE)—the final experimental approach we discuss here—on the other hand. The former have in common that the experiments are usually carried out in classrooms (or appropriate substitutes) in the communities where the subjects are located. These experiments have a short and clear-cut time frame: Subjects usually take part in

multi-round games or complete a battery of decision tasks and leave the experimental site afterwards.

For FE, which initially were a tool to evaluate policy interventions, this is vastly different. FE are also called randomized controlled trials (RCT), and we use these terms interchangeably. RCT usually imply a costly large-scale intervention changing specific parameters, e.g. with respect to health (e.g. de-worming programs) or education (e.g. provision of text books) that affect people’s daily life. They usually last for several weeks, months, or even years. Detailed socio-economic data of the participants are collected for subsequent impact analyses. A typical example is the much cited randomized evaluation of the conditional cash transfer program PROGRESA in Mexico (Skoufias, 2001). However, more recently RCT graduated from mere impact evaluation approaches to bespoke tools designed to study fundamental problems in economics such as the role of incentives, social learning or inconsistent time preference (Duflo, 2006).

One major advantage of RCT is that they fully reflect the complexity of decisions and interactions in the real world. For example, under field conditions decisions may be made at household or group level (e.g. in a firm), involving complex bargaining processes, which are difficult to simulate in the laboratory (Harrison and List, 2004).