Evaluation of the construction model against the iden- iden-tified research gapiden-tified research gap

For the evaluation of the artefact - the developed construction model - against the identified research gap, the focus is on the correctness of its construction based on requirements defined in the forefront. For this evaluation exist different approaches as listed in table 4.2.

Table 4.2: Systematization of evaluation approaches for the evaluation against the identified research gap [Riege et al.,2009]

Evaluation Approach Exemplary Utilization

Demonstration Example vom Brocke[2006];Gehlert[2007]; Klesse[2007]

Prototype Construction vom Brocke[2006];Gehlert[2007]; Braun[2007]

Prototype Application Braun[2007]

Attribute based Comparison Gehlert[2007];Braun[2007];Klesse[2007]

Meta-model based Comparison Braun[2007]

Simulation König and Weitzel[2003]

Survey Gemino and Wand[2003];Klesse et al.[2005]

Laboratory Experiment Batra et al.[1990]

Field Experiment Braun[2007]

Action based Research Grütter et al.[1998];Schwinn[2006]

For the evaluation against the research gap in the thesis at hand, anattribute-based com-parison and thesurvey-based comparison could be used. The other evaluation methods cannot be used primary due to the character of the model to be evaluated. A construc-tion model is not an artefact which can necessarily applied in the practical environment.

Therefore, approaches as field experiments or action based research cannot be applied.

They are originally used to evaluate resulting artefacts instead of models in terms of construction processes which lead to artefacts.¹⁹

19Reasons why the other approaches cannot be used: Demonstration Example: The application of the developed artefact at a fictional company is not suitable as it is a process model which has no company-relation. Theconstruction or application of a prototype is not pursued as a software-based implementation of the construction process is not in the focus of this thesis. Simulations cannot be applied since the application of the process model cannot be simulated. Asurvey could hardly be used as the respondents might not necessarily be aware of the criteria of correctness. Alaboratory experimentdoes not play a role as the creation of laboratory conditions is challenging for construction models, comparable to thedemonstration example.

Consequently, for the evaluation against the research gap, the attribute based compar-ison is selected, as already published sets of established attributes for the evaluation exist.

The attribute-based evaluation in this research consists of two parts. As a first step, the developed construction model is tested against the criteria for design science research by Hevner et al. [2004]. Those criteria act as a basis for the model construction and there-fore represent the mentioned pre-defined requirements.²⁰ The results of the evaluation based on the criteria by Hevner et al.[2004] can be found in section 4.3.1.

The second part of the evaluation against the research gap is oriented towards Becker et al.[1995] principles of general accepted modelling. Those principles have been devel-oped to improve the quality of models in the field of information systems. The results of this evaluation can be found in section4.3.2.

4.3.1 Evaluation against the identified research gap - The principles of Design Science Research

For the first part of the evaluation against the identified research gap, the developed construction model is evaluated based on the principles of design science research by Hevner et al.[2004] in order to test the construction model for its theoretical foundation.

The principles have been developed originally in order to act as guidelines for research in Information Systems. The model by Becker et al. [2009] - which has been taken as a basis of the construction model presented in this chapter - is already evaluated based on these principles. Therefore only those aspects of the construction model at hand are re-evaluated, that are different to the model by Becker et al. [2009].

• Iterative development procedure

In the course of the construction, several evolutionary stages of the model have been developed, executing multiple testing and fitting phases, based on the discus-sion with the members of the focus group, the industry experts. Their potential

20Riege et al.[2009] do not mention these criteria in the list of evaluation methods. They are focusing in their systematization of evaluation approaches on the European Information System research instead of the Anglo American understanding of information system research and mention numerous"significant differences" between those two fields. At the same time, it is stated that those are not relevant for the aspect of evaluation. Therefore, the approach byHevner et al. [2004] with an Anglo American background, can be applied in this thesis, which belongs to the European Information System Research.

negative evaluation and the resulting rejection can result in a loop back to an ear-lier construction phase until the model is approved. This re-iteration assures that only such a model is finally documented, which has been completely approved. In addition, as a potential first iteration already exists at stage four, potential errors are not carried through the whole model construction.

By cooperating with different industry experts from different companies, the dan-ger of a one-sided influence on the model design is reduced [Yousuf,2007].

• Model evaluation

The model is tested multiple times and evaluated based on the use of a focus group and an industry deployment phase, with both the initial model as well the fitted model being tested. By comparing the model’s maturity evaluation of different companies with industry experts’ evaluations, the realistic representation and understanding of real-world maturity in the model is evaluated.

• Multi-methodological procedure

Several research methods can be applied in the course of the construction process, such as quantitative methods in terms of text mining for the identification of Big Data dimensions, and quantitative approaches from the field of test theory for the model population. Additionally, qualitative research methods as expert interviews are applied during the model evaluation phases.

• Identification of problem relevance

The different maturity levels of Big Data and related capabilities are of interest to both scientists and practitioners. Maturity models as a sub-form of reference models are developed to support the design of information systems. Therefore, they can have an influence on managerial decision-making, e.g. for the development of organizational structures, budget allocation etc. As this managerial decision making is located at the beginning of a change process, e.g. towards a more data driven company, the ex-ante capability evaluation using a maturity model gains in relevance. Those aspects have to be worked out in the beginning of the model construction as an early justification of the model building.

The remaining aspects byHevner et al.[2004] (Comparison with existing maturity mod-els, problem definition, targeted representation of results,and scientific documentation)

are not affected by the changes made to the model byBecker et al.[2009] and therefore do not have to be re-evaluated.

4.3.2 Evaluation against the research gap - The principles of general accepted modelling

Besides the evaluation based on the principles of design science research (section4.3.1), the construction processes are tested for coherence with the principles of general accepted modelling. These belong to the the field of business information system engineering, derived by Becker et al. [1995] from the generally accepted accounting principles [Leff-son,1987]. The contained conventions act as guidelines during the modelling process, intended to improve the model quality.

The evaluation of (process-) models based on the principles stated above has been car-ried out numerous times and therefore has been established as a community standard [Schütte,1998].

The six normative oriented principles (table4.3) have been developed as a basis for the evaluation of the model construction and hold a "[...] customer oriented understanding of model quality." Customers in this case are individuals that will apply the construc-tion model for the development of a maturity model, which can be both scientists and practitioners.

The six principles cannot be applied completely as the construction model - developed in this chapter - differs in a number of aspects from the understanding of models in the field of business information system engineering:

• Lack of an overall context that the model has to be placed in: As the model is neither part of a general company or process model, nor is it supposed to be compared with other construction models, the evaluation aspects targetingComparability do not have to be taken further into account. The same accounts for the aspect of Profitability. Model refinements do not have to be evaluated in the forefront regarding the potential resulting benefit, as the model is not applied in a business context with a financial goal. The aspect for profitability becomes more relevant, when the focus is on the model application.

Table 4.3: Principles of general accepted modelling [Becker et al.,1995]

Criterion Description

Correctness i) Syntactic correctness: Compliance with the rules of the modeling language ii) Semantic correctness: Consen-sus amongst model users regarding the correctness of the overall model as well as individual parts of the model.

Relevance The focus is on the modelling of only those circumstances that are relevant for the underlying modelling purpose.

Profitability Determination of degree of the model refinement depend-ing on the relation between the use and costs of the ad-ditional refinement.

Clarity Aiming at the readability, clearness, and understandabil-ity of the model by focusing on appropriate hierarchy, appropriate layout, and receiver-oriented filtering.

Comparability i) Processes in the real world which are perceived iden-tically should be ideniden-tically modelled as well. ii) New, company-oriented models should be set-up on similar constructs, using the existing company-oriented models in order to foster the meta model transformation.

Systematic Structure Identical objects, utilized in different models (data model, process model, etc.), should be used correspondingly in order to achieve consistency of the overall model.

• Overall context: The Systematic Structure is not relevant as the construction model is not part of an overall model, therefore a consistent use of notations between different models does not have to be pursued.

After eliminating those aspects due to the different type of model in focus, the remaining aspects are Correctness, Clarity, and Relevance.²¹ These aspects contribute to an understandable description of the individual process steps including the resulting arte-facts. The demand of each criterion and how it is fulfilled by the developed construction model will be explained in the next step:

The aspect of Correctness is targeting the syntactic and semantic correctness. Al-though no underlying modelling language in terms of e.g. EPKs exists, the description

21In contrast to the work by Mettler [2010], the construction process model instead of the resulting maturity model is evaluated in the context of the evaluation against the identified research gap. The author is convinced that the construction model, despite its differences as explained before, can be understood as a process model as targeted by the principles of general accepted modelling. This does not account for the resulting maturity model, as it does not follow a process logic.

of the construction process is based, comparable to Becker et al.[2009], on the notation DIN 66001, fostering the overall correctness of the model.

The aspect of Clarity is contributing as well to the understandability of the model. A sufficient readability, clearness, and understandability have been achieved, as each step has a distinct name and the related actions for each step are documented, containing both the needed input and the potential output as shown in figure4.1.

Relevance is focusing on the steps, which are taken into account for the model. The demand is that only those aspects should be integrated in the model, which are relevant for the underlying modeling purpose.

The demand is fulfilled, as in case one of the construction steps would have been left out during the construction, the construction of the maturity model would not be possible.

In order to clarify the relevance, each step and its contribution to the model development has been explained in detail in this chapter.

Altogether, the presented construction model follows the principles of design science research and follows the principles of general accepted modelling. The model has been evaluated successfully against the identified research gap and tested for the correctness of construction. Therefore, the construction model represents a contribution to design science research [Hevner et al.,2004] and will be applied for the model construction in Chapter 5.