After having established the context of this thesis, as well as the problem and motivation, the following sections look at a suitable research methodology to address the issues outlined earlier.
This thesis examines the design and evaluation of methodologies and support systems for IIoT project managers. The chosen research methodology must be one that
specifically supports the creation and evaluation of new artifacts. For this purpose, design science is examined as a suitable methodology in the following. Furthermore, to strengthen the evaluation aspect of this thesis, case study research is also considered.
1.6.1 Design Science Research Methodology for Information Systems Research
In 1991, Nunamaker Jr et al. (1990) already introduced the idea of integrating system development into the research process. Simon (1996) envisions design science as a pragmatic research paradigm that enables the creation of innovative artifacts to address real-world problems.
Six years after the initial paper from Hevner et al. (2004), Hevner and Chatterjee (2010) describe design science as “an effective means of addressing the relevancy gap that has plagued academic research, particularly in the management and information systems disciplines.” They argue that natural science research methods are more “appropriate for the study of existing and emergent phenomena,” while problems that require creative, novel, and innovative solutions “are more effectively addressed using the type of paradigm shift offered by design science.”
Peffers et al. (2007) describe principles, practices, and procedures to carry out design-science-oriented research. The design science process described here includes six steps:
problem identification and motivation, definition of the solution objectives, design and development, demonstration, evaluation, and communication. Peffers et al. (2007) also describe different possible research entry points for each of these steps, as can be seen in Figure 11. These entry points include problem-centered initiation, objective-centered initiation, design and development-centered initiation, and client/context initiated.
Figure 11: Design science research methodology for information systems research (Peffers et al., 2007)
Peffers et al. (2012) have performed an evaluation of design sciences artifacts based on the evaluation of 148 design science research articles published in selected journals. They analyze these articles to develop taxonomies of design science artifact types, as well as methods for artifact evaluation. Based on this, they then provide two studies. The first one focuses on “instantiation evaluated by prototype,” where instantiation is an artifact type, and prototype is an evaluation method type. The second example study focuses on
“method evaluated by case study.” They conclude, “The case study lends itself for use in evaluating the efficacy of a designed object that is intended to be used in a complex organizational setting where a simple experiment or other simple test could not be used to adequately show the efficacy or performance of the object.” Consequently, the following section looks at case study research in more detail.
1.6.2 Case Study Research
Benbasat et al. (1987), Robson (2002), and Yin (2003) all provide widely cited definitions of a case study, agreeing that it is an empirical method aimed at
“investigating contemporary phenomena in their context.”
Robson (2002) calls it a research strategy, emphasizing the need to use multiple sources of evidence. Yin (2003) notes that the “boundary between the phenomenon and its context may be unclear.”
Easterbrook et al. (2008) make an interesting differentiation between exploratory and confirmatory case studies.
Runeson and Höst (2009) define a case study research process, which includes five major process steps: case study design, preparation for data collection, collection of evidence, analysis of collected data, and, finally, reporting (see Figure 12).
Figure 12: Case study research process
According to Runeson and Höst (2009), case study design includes the definition of the case, the case study protocol, and ethical considerations. The case study design must clearly define the case and its unit of analysis. Case study objectives, hypotheses, preliminary research questions, and theoretical bases should be clearly defined. The case should be adequately defined in terms of size, domain, process, and subjects.
Considerations include: Are data triangulation (multiple sources) and method triangulation (multiple methods) reflected in the design? How clear is the rationale behind the selection of subjects, roles, artifacts, and viewpoints?
After the initial case study design, Runeson and Höst (2009) describe the process of data collection, including first-, second-, and third-degree data sources, interviews, observations, archival data, metrics, and checklists.
In data analysis, Runeson and Höst (2009) differentiate between quantitative data analysis and qualitative data analysis. For qualitative data, they differentiate between hypothesis-generating techniques and hypothesis-confirmation techniques, based on Seaman (1999). Hypothesis-generating techniques can be used for exploratory case studies, while hypothesis-confirmation techniques are used for explanatory case studies.
Triangulation and replication are cited as examples of hypothesis-confirmation methods, based on Seaman (1999). Another important factor is the validity of a study since it directly related to the trustworthiness of the results.
Finally, the reporting communicates the findings of the study. Jedlitschka and Pfahl (2005) propose guidelines for reporting of experiments that have been evaluated by Kitchenham et al. (2008). An important point in the proposed reporting standards is cross-study comparisons through systematic reviews. Since case studies are usually based on qualitative data, the level of standardization here is lower.
1.6.3 Approach Taken by This Thesis
Given the positive evaluation of both design science (for methodology design and evaluation) and case study research (for evaluation specifically), this thesis combines both approaches. For design science, the approach from Peffers et al. (2007) was chosen. For case study research, the process outlined in Runeson and Höst (2009) was chosen. The following analysis shows that in the context of the goals of this thesis, the two provide a good foundation.
Starting with the process outlined in Peffers et al. (2007), the required iterations and phases for this research project were mapped to it (see Figure 13).
Figure 13: Research methodology and approach taken by this thesis This research project includes the initial process iteration, plus a second, incremental iteration added to incorporate results from the first iteration and to provide a deep dive on selected topics that emerged as important in the first iteration.
Phase one of this research project is the initiation phase, looking into the problem definition and motivation. The matching section of this thesis provides an overview of IIoT challenges and opportunities, as well as a specific view on the challenges faced by project managers in the IIoT. It also contains an overview of the chosen research methodology and structure of this thesis.
Iteration one, phase two looks at solution objectives. This includes a definition of the focus areas of this project, reuse of existing artifacts, a concrete example, and
assumptions and constraints.
Iteration one, phase three considers the concrete design and development of the solution, which has been named IgniteWorx. The design and development of
IgniteWorx has two main aspects: design and development of concrete content for the system, as well as design and development of the supporting IT solution.
It should be noted that while iteration one, phase four (demonstration) looks at a concrete reference implementation based on the concepts developed in this thesis, this implementation is actually not in scope of the thesis itself. Rather, it is an
implementation done by the IIC based on the open-sourced design concepts created as part of this thesis (see copyright overview, page 44).
Iteration one, phase five is an evaluation based on an embedded case study (Yin, 2003).
In this case study, four different projects are evaluated against the IgniteWorx framework defined in this thesis.
Iteration two performs an incremental enhancement of the results of iteration one, focusing specifically on the issue of capturing user feedback. This includes a discussion of the objectives of this incremental enhancement (6), the design and development (7), the demonstration (8), and finally a simulation-based evaluation of the enhancements (9).
Finally, the thesis ends with a general evaluation of the process and the key findings, as well as an outlook into potential future work.