Level of analysis

Overcondence is an individual characteristic. However, based on the ndings dis-cussed in Chapter 4, the impact of individual overcondence on innovation-evaluation at the group level can be driven by two interrelated domains: the impact of overcon-dence on overcondent subjects' individual actions and the interaction of overcondent subjects with other participants.

First, the characteristics and inuence of overcondence can be observed on an indi-vidual level. Overcondence is a trait that nds expression via overcondent subjects' actions. Overcondence acts upon decision making particularly in the context of inno-vation evaluation. Condence appears to be variably pronounced among individuals who participate in innovation evaluation tasks. While mild overcondence in one's own personal ability in evaluation tasks has been found to be a very common human trait by early psychological studies (Fischho et al. 1977), considerably higher and more het-erogeneous degrees of overcondence appear to be particularly present in innovation evaluation tasks (Hayward et al. 2006). We presented arguments in the previous section that self-selection may systematically attract some individuals with particularly high degrees of overcondence, such as entrepreneurs, inventors and top-managers, to par-ticipate in innovation tasks. The previous chapter showed that the degree of individual overcondence inuences individual evaluation of innovations, but we also stressed that an understanding of this phenomenon in the context of information markets is lack-ing. Hence, the rst level of analysis in our study addresses the genesis of individual overcondence in innovation evaluation and its impact on individual decision making in innovation evaluation via information markets.

Second, overcondent subjects interact in information markets with other participants.

On the one hand, overcondence could potentially inuence the trading by overcondent

subjects, which would then be perceived, processed and incorporated by other partici-pants into their actions. On the other hand, overcondence could potentially inuence the way overcondent subjects perceive other participants' behavior in the market. For example, as discussed in the previous chapter, overcondent subjects may be less open to market signals to update private information (Deaves et al. 2009). Both directions of interaction can be studied by uniting multiple subjects in information markets, which leads to the second level of analysis.

In summary, to build an understanding about the impact of overcondence on the indi-vidual level, yet also gain valuable insight about its impact on the quality of information markets for innovation evaluation, we need to study the particular inuence of overcon-dence on (1) individual behavior and (2) interactions in information markets. Only then can we gain a true understanding about the relationship between overcondence, individual action and information market outcomes.

Our empirical studies therefore need to focus on two distinct levels of analysis. On the rst level, we will explore the impact of overcondence on individual behavior in infor-mation markets in a controlled individual environment. On the second level, we will investigate the impact of overcondence on the evaluation quality of an information market in an integrated environment using individual agents but group-based outcomes.

5.3. Methodology

We focus our study on the impact of the individual characteristic of overcondence on both individual behavior and the group-level outcome (i.e. evaluation quality) of infor-mation markets. Sorensen et al. (2010) show that surveys, eld studies, and case studies have traditionally been the methods of choice for innovation management researchers, although researchers have stressed that laboratory experiments would provide more valu-able results if the research object [were] practically and meaningfully isolated from a broader context of innovation (Sorensen et al. 2010). In our research, overcondence acts as the independent variable, while the dependent variables are individual behavior in information markets and innovation evaluation quality. Our research object therefore ts the abovementioned suggestion for applying laboratory experiments.

In psychology, causal relationships between traits and behavior are predominantly

stud-ied via laboratory experiments (Shaughnessey et al. 2012). It has been suggested that similar research questions in the eld of management decision making should employ the same methods as psychology, although experiments could suer from low contextual realism (Scandura and Williams 2000). Laboratory experiments can provide signicant benets in answering our research questions.

However, signicant practical barriers oppose eld studies. Our subjects cannot be easily sampled or recruited for eld experiments. Furthermore, our focal characteristic, overcondence, is heterogeneously dispersed and is not easily observed among individuals (Anderson and Kildu 2009). Additionally, overcondence appears concentrated among individuals like CEOs, venture capitalists and entrepreneurs, who presumably have rela-tively little time to participate in extensive studies that focus on their individual behavior and interactions in group tasks. Furthermore, overcondent yet high-ranking subjects may be more reluctant to participate in experiments that could ultimately undermine their decision-making competency.

More importantly, focusing on the scientic methodology, we can only draw inferences about causal relationships between condence and individual actions if we successfully control for condence levels among subjects. The strength of laboratory experiments lies in the rigid control over variables and environmental conditions in order to draw conclusions about such causal relationships (Willer and Walker 2007). The systematic manipulation of overcondence in a laboratory experiment produces three advantages for our research approach.

First, laboratory experiments allow the recruitment of homogeneous subjects in or-der to reduce variance in characteristics other than the manipulated feature. However, it should be noted that a large degree of variety in subject characteristics provides addi-tional variance and unwanted error when relating overcondence to individual behavior and information-market outcomes.

Second, laboratory experiments allow researchers to exert the best amount of con-trol over the desired independent variable. A successful manipulation creates variance in individual condence that allows the valid investigation of the relationship between (over)condence and behavior in information markets. Aside from its practical barriers, manipulating condence outside the laboratory may likely be impossible be-cause individuals are highly inuenced by the context and cues given in previous tasks.

Experiences of task performance within their natural environments are embedded within individuals, which likely reduces the potential eect of condence manipulation.

Third, laboratory experiments can create a close relationship between the source

of overcondence and the focal task, in which overcondence inuences subject behavior. Research has shown that individual levels of overcondence do not necessarily span domains. Being overcondent in one task does not reliably predict overcondence levels in another task. Perceived task diculty and social context largely predict the rel-evant facets of overcondence. Tasks that subjects perceive to be dicult for themselves but even harder for others likely produce the highest amount of overcondence, both in an absolute sense and in relation to others (Larrick et al. 2007). To induce the least amount variance when studying overcondence in innovation evaluation tasks, it may therefore be important to manipulate overcondence via a close relationship between the source of overcondence and the task domain.

Overall, a critical assumption underlying the interpretation of laboratory experiments is that the insights gained in the lab can be generalized to the world beyond.

For physical laws and processes, evidence supports the idea that what happens in the laboratory is equally valid in the broader world. Yet much controversy exists regarding the conditions in which the same will hold true for social experiments that focus on individual psychological conditions or the behavior of human groups. Levitt and List (2007) highlight that moral and ethical considerations, the nature and extent of scrutiny of ones actions by others, (self-)selection processes, the context of decision making, and the stakes of decision making heavily inuence behavior in the lab, apart from monetary considerations. The researchers stress that these factors dier between the laboratory and the social context the lab aims to resemble, which could prevent researcher from being able to generalize ndings in a straightforward manner.

Other researchers have discussed recruiting student subjects in particular. Recruiting students brings the advantage that the recruiting process does not require extant re-sources, as subjects are readily available and usually cheap. Another advantage is that student characteristics can be better controlled in terms of educational background and domain knowledge, compared to drawing subjects from the general population. Thus, results would be more closely related to the experimental manipulation since they would exhibit less unexplained error. Additionally, carefully selected subjects may be more appropriate than random samples in cases where the experimental results are to be re-lated to professional groups that can hardly be recruited for experiments. For example, accounting MBA students have been found to be appropriate subjects for studying the behavior of professionals, while bachelor students in economics did not suciently resem-ble accountants' behaviors in various experimental tasks (Liyanarachchi 2007). Similar ndings have also been reported for studies related to the underlying research of

group-based innovation evaluation, e.g. the nding that undergraduate students should not be used to study the behavior of managers in decision support systems but that participants can be appropriately sampled from graduate students (Remus 1989). As a consequence, recruiting students may uncover systematic error if researchers falsely assume the gen-eralizability of the student behavior in the experiment to the real-world behavior that the researcher aims to explain.

In the end, Levitt and List (2007) emphasize the exclusive value of laboratory exper-iments that study economic behavior, regardless of the abovementioned shortcomings.

Instead of using experimental results as highly diagnostic cues for real-life behavior, the authors suggest to understand them as a crucial rst understanding regarding fundamen-tal behavioral mechanisms that would otherwise be dicult to observe and understand in real-world studies. Our experiments build upon this suggestion.

Our empirical studies address the impact of the individual characteristic of overcon-dence on individual behavior in information markets and on innovation evaluation quality as a group-based outcome of information market trading. Separate experi-ments allow us to isolate these two levels of analysis and their respective goals to reach a more valid conclusion regarding the impact of overcondence on each dependent vari-able. As our analysis units are individual behavior in information markets and outcomes of information markets, we selected individual and market-related outcomes as granular dependent variables.

On the individual level, we needed to develop a controlled environment that allowed us to isolate and investigate the relationship between overcondence and individual behav-ior. On the level of information market outcomes, we needed to test information market performance in the light of real innovation evaluation tasks to validly assess the impact of overcondence on information market ecacy.

Finally, it would be dicult to recruit a sucient number of subjects when focusing on outcomes of single market periods as the granular level of analysis. Each market period characterizes only case for both levels of analysis.

Many experiments in behavioral market economics have encountered such a subject-related bottleneck by allowing subjects to participate in more than one market period while still analyzing each period independently (e.g. Seybert and Bloomeld (2009), Healy et al. (2010), and Jian and Sami (2011)). Within-subject designs are gener-ally accepted in psychological and economic experimental research, but researchers have

been encouraged to better account for subject-based errors during analysis. Methodolog-ical work has stressed the importance of applying multi-level approaches when analysis requires controlling for within-subject error (Judd et al. 2001) or aims to explain group-level outcomes (Kashy and Kenny 2000; Van Kleef et al. 2010).

To summarize, our empirical studies use laboratory experiments to investigate the im-pact of overcondence in innovation evaluation by drawing from student subjects. One experimental study is focused on individual behavior in information markets, while the other experiment is focused on the impact of overcondence on the outcome of informa-tion markets for innovainforma-tion evaluainforma-tion. Subjects interact in multiple market periods, a condition which has been successfully controlled for by using within-subject designs to analyze the results.