Limitations

Concerning the first part of the model evaluation, it could be shown that a quantitative approach can be applied for the assignment of items on different maturity levels in the developed model construction context. At the same time, the discussion of the initial model with the focus group members - as the first step of the evaluation against the real world - yet revealed the need for a further fitting.

Every item had to be reassigned by the industry experts by an average of 0.77 levels.

This low reassignment shows that although all items had to be reassigned, the applica-tion of the quantitative approach leads to suitable results. It could be shown that the extent of reassignment is not linked with the novelty of the topic in focus. Both items related to the established topic known from the field of BI, and novel topics underwent similar reassignments. Similarly, within the group of topics with a low/high extend of reassignment, both novel as well as established topics could been found.

The model could be successfully deployed in an industry context as the second part of the evaluation. In this context - the identification of companies, covering the whole range from maturity level one to maturity level six in order to achieve a full-range model evaluation - has proofed itself as demanding for a novel topic like Big Data.

Nonetheless, a high agreement between the model based evaluation and the expert’s evaluation of companies maturity could be found. This can be interpreted as an adequate representation of the practical understanding of maturity by the developed model.

The positive model evaluation gains in weight when looking at the different scopes of the two maturity evaluations. The model is focusing on the dimensions Organisation and Data, whereas the expert is evaluating the companies’ capabilities along all Big Data dimensions. Despite these differences, the comparison shows a high agreement between the model and the experts’ based assessment, indicating a broad coverage and relevance of the selected dimensions.

6.2.1 Maturity concept based limitations

One major limitation concerns the relation between maturity and performance. A higher maturity, although perceived as a desirable state, does not necessarily lead to an im-provement of a company’s performance. An insufficient operationalization can override a high maturity. Consequently, a correlation between organizational performance and maturity cannot be drawn directly, although several topics of the developed model focus on the operationalization, e.g. the definition and standardization of analysis processes.

Therefore, the relevance and explanatory power of the developed maturity model can be fostered by examining an improved performance on a higher maturity level. A longi-tudinal study, analyzing different companies on different maturity levels and comparing financial performance indicators in the course of time could help to close this gap.

The aspect of performance is of special interest for Big Data with regard to the volume of potential investments and the needed organizational changes. Due to the novelty of this topic, to the best of the authors’ knowledge, no research in this field exists so far.

A second limitation results from the dynamic character of the topic Big Data. Due to the speed of development e.g. regarding the costs and the scope of analytics applications as well as the increasing professionalism in the utilization of large amounts of data with different degrees of structure, the model holds a high need for a regular update. With regard to the needed time for the model development in a research context, especially the needed time for data gathering and the evaluation process, it can be partly outdated when the construction process is completed. Therefore, the maintaining process plays a critical role in order to ensure a permanent model relevance.

Another potential limitation comes with the selected level of abstraction, being a core decision of each maturity model construction process. The maturity model holds a certain level of abstraction due to the generalization regarding industry, company size, and application. On the one hand, this limits the explanatory power, as the contained items are selected to be regained in the majority of the companies. With regard to the early stage of the topic Big Data, the trade-off has been made towards a general model that is intended to act as a starting point for further, more dimension specific maturity models in the Big Data context. With the increasing spread of Big Data related applications, it is assumed that with each maintain process, the granularity of the models measurements can be improved.

6.2.2 Method based limitations

Within the contextual brackets, the Design Science Research, both qualitative and quan-titative methods have been applied: mainly i) text mining, ii) focus group, and iii) IRT.

The applied methods will be evaluated regarding the resulting limitations.

Text Mining

One limitation resulting from the applied methodology can be found in Chapter2during the identification and validation of dimensions on Big Data. The limitation targets the influence of the breadth of the analyzed topic and the results of the topic models.

It could be shown that the explanatory power of the approach decreased when the analyzed corpus contained only dimension-specific publications, e.g. solely technology-oriented. The results based on the broader corpus of Big Data publications, including all dimensions, led to better results. These differences in explanatory power are represented in the different values for the model precision. Therefore, the results from the topic model application on the specific dimensions have not been processed further.

Additionally, the corpus with 247 analyzed abstracts is relatively low compared with other publications using the topic model approach on established topics, owed to the novel character of Big Data. With the increasing number of publications regarding Big Data, the explanatory power of the topic model applications may increase. As the focus of the work at hand is on the phenomena of Big Data, further search terms have been left out intentionally.

Nonetheless, for future research, the focus on the search term Big Datacarries the risk that publications, which are relevant for the topic but marked with another tag, e.g.

the rising notion of Advanced Analytics are left out. Furthermore, relevant topics, e.g.

organizational aspects might not appear in the analysis, as publications on that topic have not been published yet.

Focus Group

The input of the focus group depends amongst others on the members’ academic back-grounds and practical experience. Therefore, the composition of the focus group has an influence on the overall results, as the focus group members have an influence on the resulting maturity model in terms of the contained topics and measurements and their assignment to different maturity levels. This limitation could partly be solved.

Although a random set of focus group members is hard to be achieved, by increasing the number of the members of the focus group with a heterogeneous background, regarding to the aspects mentioned above, a broad knowledge base could be reached. Yet, a higher number of new focus group members for each construction step could contribute to the broader knowledge base.

Item Response Theory

The potential limitations that go along with the application of the IRT have already been discussed in the model population step (Chapter5). Besides that, the basic concept of using the item difficulty as a vehicle to determine the maturity of each measurement, contained in the questionnaire can be seen critical, as it leads to problems concerning the different scales. As it has been described in construction step 5 (Chapter 5), the continuous numbers per item (valuation 0-1) can be seen in conflict with the maturity concept, limited to absolute numbers, that range in the thesis at hand from 1 to 6.

Consequently, the delta of the item difficulty between the item with the highest item difficulty value and the one with the lowest difficulty of one maturity level is not nec-essarily equal along all maturity levels. This limitation loses in relevance as, resulting from the re-organization of the items based on the discussion with the focus group, the final order of the items is not following the initially calculated difficulty values.

As the approach from the IRT has been applied on a data set resulting from answered questionnaires, the Characteristics of the Respondents pose an influence and potential limitation. The responses do not necessarily reflect the actual situation of a company, as the respondent may not have a sufficient overview, e.g. about existing projects. This accounts especially for larger companies. Consequently, the background and knowledge of the respondents have a major influence on the quality of the initial model. This po-tential bias is not specific for this thesis, it is rather a general problem of data gathering based on un-controlled answering of questionnaires and can be found in similar works as well [Lahrmann et al., 2011b]. By distributing the questionnaire in different fora, allowing an anonymous answering without a personal approach, the sample has been compiled as random as possible.

For future research, personal interviews, following a random sampling of companies and interviewees for the data gathering could be used in order to gain a better insight into

the company practices and allow to interpret the responses.¹