Summary of results

The main goal of this thesis was to create a more thorough understanding about the impact of judgmental biases on the evaluation of innovation via information markets.

This goal was split in two research objectives.

In Part Two of this thesis, we addressed the rst research objective and created a conceptual background that set the scene for subsequent empirical analysis. A thorough introduction to information markets for innovation evaluation and judgmental biases prepared a common and detailed understanding of the research context. We found that judgmental biases are frequently present in innovation management decision-making and may signicantly impact the outcomes of innovation evaluation tasks. Of the most preva-lent biases identied in Section 4.3, overcondence appears to be particularly important in the context of innovation evaluation in general and information markets in particular.

From there, we determined our second research objective, which was to empirically study the impact of overcondence on the evaluation of innovation via information mar-kets. First, the empirical focus was narrowed to the potential impact of overcondence on the quality of the innovation evaluation. We dened overcondence as the individual tendency to overestimate own prediction performance overall and in relation to others.

Two experiments with information markets were conducted.

The rst experiment investigated the impact of overcondence on individual behavior in the context of information markets. We drew inferences about individual behavior by creating an information market environment that allowed us to analyze individual behavior in a controlled setting. Subjects received a treatment that manipulated their condence levels, after which they traded with a controlled articial market agent over a number of sessions. The rst experiment demonstrated that higher levels of condence

induce individuals to trade more stocks in information markets, engage in trading ear-lier, trade more in opposition to other traders' deecting beliefs, and be less likely to update personal beliefs in relation to market signals after the information market has nished. These results indicate a stronger inuence of overcondent subjects' beliefs on the aggregated information that is expressed via market prices.

The second experiment investigated the impact of overcondence on the prediction quality of information markets and their participants. Subjects received a treatment that manipulated their condence to either overcondent or low condence levels. Pairs of similarly treated subjects were brought together with two noise traders who had not received any treatment. Thus, the second experiment featured information markets in which four traders aggregated their expectations via trading. The results documented that the presence of overcondent participants may signicantly lower the prediction quality of information markets. When overcondent subjects are well-informed re-garding the underlying prediction target, they exhibit strong convictions rere-garding their interpretations of the provided information. Consequently, they trade more aggressively, as suggested by the rst experiment, and more strongly oppose other participants' mar-ket signals. This limits both participants' learning from those marmar-ket signals and ecient information aggregation. In the presence of overcondent traders, market predictions are less likely to converge to form a homogeneous signal that could be better interpreted by non-informed subjects to allow them to update private information. In markets with overcondent subjects, noise traders' predictions have a more strongly negative impact on market prediction quality and these traders will be less likely to improve their private predictions after the information markets have nished.

When overcondent subjects can acquire diagnostic information prior to the mar-ket, they will be less likely to do so, since acquiring information is less valuable to them.

Acquiring less information leads to higher individual prediction error on the part of overcondent subjects before the information markets and less information in the infor-mation market, both of which ultimately negatively impact market prediction quality.

Higher prediction errors can be related to (1) less trading activity by overcondent sub-jects because they are aware that they are less likely to acquire information and thus less informed and (2) less informed trading activity by overcondent subjects if they ignore the fact that they are less informed because they see less value in acquiring information.

The two experiments expand the current state of research with regard to judgmental

biases in the context of information markets. The results of the rst experiment provide a detailed picture of the impact of overcondence on individual behavior in a controlled laboratory experiment. The second experiment demonstrates that the presence of over-condent subjects will systematically lead to lower prediction quality in information markets in the given informational environments.

9.2. Limitations

Behavioral experiments are narrowly focused with regard to incentives, participants, inter-subject relationships, and experimental environments to allow the investigation of causal relationships while controlling for potential interference. Such rigorous control over research conditions produces numerous limitations, which potentially reduce the extent to which the ndings can be applied to relationships in related experimental or real-world contexts. Thus, it is important to investigate the limitations of the present experimental research for two reasons. First, awareness about the limitations of an experiment provides the necessary consideration for developing theoretical implications from the results. Researchers are stressed not to overly generalize from their ndings.

Second and closely related to the rst point, theoretical assumptions to which empirical results may appear ambiguous provoke suggestions for future research to increase the ndings' reliability and obtain a clearer understanding about the underlying mechanisms that produce the phenomena in question.

First, all of our experimental subjects were drawn from a body of master stu-dents in engineering disciplines. Our sampling therefore partly confounds our inference from the current body of research that overcondence is most likely to be present among entrepreneurs, innovation managers, and executives. While entrepreneurs and managers in innovative technology companies may be more likely to possess engineering degrees than the general population, certainly not all the students in our sample will join these ranks. However, we addressed this issue by developing an experimental treatment that allowed us to create overcondence articially. On the one hand, such an experimental treatment comes with a signicant benet. Overcondent subjects can be allocated in any fashion that the experiments require. On the other hand, such treatment may lack external validity if the articially-induced overcondence diers from the natural over-condence among the above-mentioned groups. While our experiments showed that the behavior by subjects with induced overcondence resembled the behavior of untreated

student subjects with natural overcondence, we cannot draw inferences about a similar resemblance to the behavior of overcondent entrepreneurs, inventors, innovation man-agers, or senior leadership.

Second, there may be dierences between overcondence as induced by our treatment and natural overcondence. However, we did not nd signicant dier-ences in the dependent variable means when comparing naturally overcondent individu-als have a strong better-than-average perception in our control group and participants in the treatment group. Second, we had to rely on students as subjects for our experiment and could not recruit real managers, engineers or blue-collar workers from automobile companies. However, we have put forward arguments for why we still believe our manip-ulated subjects felt suciently involved in the industry to make informed statements.

We believe that the eects we found would be even stronger among real decision-makers, as they would be able to tap a larger pool of private information when evaluating inno-vations.

Third, we aimed at creating relevant innovation evaluation tasks by using real-world evaluation examples. However, our innovation tasks dealt mainly with innovations that were already entering the market. Therefore, we can only assume that sim-ilar patterns would arise when evaluating at the fuzzy front end of innovations, e.g.

during the idea evaluation process. Yet, drawing from current research, we believe that overcondence may have a far greater impact on individual behavior at the beginning of innovation endeavors, when context is less understood and uncertainty considerably higher (Fowler and Johnson 2011). Alas, it would be very dicult to validate the eval-uation outcome of ideas, since a true underlying value would not be available any time soon.

Fourth, the incentives in our experiments were xed at amounts that - in the case of win-ning one of the vouchers - meant signicant nancial gains to our participants in relation to the time the experiments lasted. The average student salary at universities in the German state of Hamburg wase8.49 per hour at the time of the experiments and poten-tial winnings amounted toe20 ore50. Therefore, our experiments should have provided relevant extrinsic incentives for truthful revelation of beliefs. Yet of course, real-world information markets may dier strongly with regard to incentives. Frequently, the relationship between potential monetary awards and regular participants' salaries is much higher in corporate information markets for evaluation of innovations. For

example, the largest German telecommunications company planned to provide a pink Volkswagen convertible as a prize to spur female participation¹. However, researchers have frequently stressed that motivation-based incentives (for having better predictions than others and achieving the higher portfolio value) may be a suciently strong driver for eective participation (Dahan et al. 2010).

Additionally, our results only allow limited inferences about how overcondence may impact information markets that require subjects to invest private funds. While the additional experimental setting of the second experiment may provide some indication of how overcondence interacts with the risk of losing funds, the subjects only risked winning less rather than leaving the experiment with fewer assets than they had pos-sessed before starting.

Fifth, our experimental environment requires critical examination regarding to what extent it resembles real information markets for innovation evaluation. Any dierences may need closer examination to increase reliability when making inferences about real-world information markets. To begin with, our innovation evaluation tasks were all concerning the future market success of innovations that had recently been introduced to the market. We have presented arguments for why we choose such tasks and under which circumstances we may expect similar results if the tasks concerned early stage innovation ideas or concepts. Yet our experiments cannot provide empirical support to these arguments. Closely related to the tasks, the information cues provided during the second experiment were homogeneous among participants (if they acquired equal amounts of information). Our experiment, therefore, did not investigate how the pres-ence of overcondent subjects with very heterogeneous knowledge (e.g., marketing and R&D managers) impact prediction results. While the results of the second experiment indicate the lack of consensus building and its negative consequences on noise traders' prediction formation, we did not vary the heterogeneity of information systematically to provide further insight. Also, subjects where paid out in our experiment only after the real values materialized. As our information markets dealt with predictions of fu-ture market shares, there was a signicant time gap between participation and payout.

Hence, our experiments did not answer the question how success (or the lack thereof) by overcondent individuals over the course of multiple rounds of payout might impact their behavior and inuence in information markets. The results of our experiments should therefore be specically regarded as discussing the impact of overcondence on the

re-1The authors learned this from a personal talk with the company's project leader.