How Secondary Data Analyses Can Reduce Risk of Bias

In the previous paragraph, I pointed out how secondary data analyses can increase theoretical coherence, addressing the first part of the guiding question of the dissertation project.

The second part concerns the reduction of risk-of-bias in meta-scientific work. My colleagues and I found that secondary data analyses based on raw data can indeed reduce risk-of-bias. I will first discuss the most pressing concern: publication bias. Typically, there is “publication

pressure” on quite specific parts of a statistical analysis. More often than not, this is the p-value

for the null hypothesis significance test (NHST) testing the main hypothesis. With secondary

data analysis as I define it, the meta-analyst aggregates correlations that were usually not focal in

the original study. For example, in paper 2 of part II most studies were not concerned with sex

drive directly, but rather relationship dynamics in romantic dyads, the role of testosterone in

GENERAL DISCUSSION 21

sexuality, evolutionary strategies, or something else entirely. Thus, whether or not gender and sex drive were correlated likely had no bearing on whether or not the study would be published.

Thus, with regard to our goal, there was no reason to expect any direct publication bias at all. To be sure, there was publication bias on other values in the manuscripts. I do not mean to argue that the data base was entirely unbiased and representative. Yet, I would expect these distorting influences that are unrelated to the focal question to cancel out with regard to the correlation we were interested in. This was quite a revelation, to have a large data base and no pressing reason to question its validity. In many ways, the biasing processes for publication bias and p-hacking, the second important class of bias we considered in paper 2 of part I, are quite similar.

Publication bias is goal directed selection of studies and manuscript, while p-hacking often involves goal directed selection of data points, variables, data preprocessing steps, or analytical approaches. Consequently, the same advantages of the secondary data analysis approach apply.

Again, for the example of paper 2 of part II, the original researchers were not interested in the correlation between sex drive and gender, and thus likely did not (consciously or unconsciously) manipulate their analyses to minimize or maximize this correlation. Beyond publication bias and p-hacking, there were also some other bias-reducing advantages of the secondary data analysis approach. For one, having access to the raw data makes it possible to run meta-analytic

psychometric analyses. In paper 1 of part II, we examined the data for range restriction and unreliability. In paper 2 of part II, we were able to conduct a meta-analytic test of convergent validity for the measures we included.

In summary, we found that secondary data analysis can yield meta-analytic analyses that

have high theoretical coherence and low risk of bias. In the next section of the discussion, I

GENERAL DISCUSSION 22

would like to broaden the scope and attempt to situate the approach of secondary data analysis as described above in the grander scheme of psychological research methodology.

Integration versus Innovation in Psychological Research

When I was first introduced to psychological research, my impression was that the field proceeds in a somewhat linear, cumulative fashion. There are solid foundations, and new research constitutes more and more intricate additions that extend and ornament what is already there. Today, I sometimes feel like everyone is just tossing new rocks on a huge pile of rubble.

The new stuff buries the old stuff and walking up the pile is unsafe and strenuous, leaving one wishing for stairs. Of course, this is an outrageous oversimplification and overgeneralization, and does not do justice to the many subfields that do solid, cumulative work. But I do think that the underlying question of how the field prioritizes innovation versus integration is valid and

important. To quickly define the terms, I would draw on a view of psychological science in terms of a graph model (Koller & Friedman, 2009), where psychological variables and theories are nodes and connections between theories and variables are edges (see Figure 1). I would consider innovation to mean adding nodes to the model. Conversely, integration is A) adding edges, that is, (empirical or theoretical) connections between the nodes to the model, or B) reshaping nodes so that connections can even be made in the first place. This is what my colleagues and I aimed for in part II of the dissertation project. In paper 2 of part II, we connected gender and sex drive.

In order to do that, we had to develop a new theory of sex drive because there was no agreement in the literature (thus reshaping the ‘sex drive’ node; we obviously did not invent the concept).

The theory added further connections, tying sex drive to personality theory (Fleeson &

Jayawickreme, 2015; McCrae & Costa, 2003) and foundational psychological concepts like

cognition, emotion, and behavior. In the competitive space that is the academic job market,

GENERAL DISCUSSION 23

innovation is highly prized and is essentially a requirement for career advancement. Researchers need their own theories, and as the saying goes, theories are like toothbrushes—no

self-respecting person wants to use anyone else’s (Mischel, 2008). In this way, the psyche is carved up into ever smaller bits that generate a research niche, and research proceeds in parallel lines, carefully avoiding to trespass someone else`s territory. This is understandable, but unsatisfying.

Integration and cooperation are needed if the field is to make progress. Working on integration also does not mean foregoing theory. On the contrary, theoretical work is often required to enable new connections (i.e., theories and variables need to be made ‘connectable’). New connections themselves can be of theoretical nature, and new empirical connections can be inspired by theory. I would appreciate to see more emphasis on such integrative work in the future.

Figure 1

A Graph-based Model of Psychological Science

GENERAL DISCUSSION 24

Note: Circles (nodes) represent psychological theories or variables. Lines between the circles

(edges) represent connections between variables or theories.