KonSearch Usability Study
Evaluation of the new literature search engine of the University of Konstanz
Helena Luca
Graduate Library Trainee
Translated by Atlas Translations
October 2011
Konstanzer Online-Publikations-System (KOPS)
Abstract
This report describes the background, methods and results of a usability evaluation, which was conducted in the summer semester of 2011 in the course of the introduction of the KonSearch literature search engine at the University of Konstanz. The new search engine is based on the Resource Discovery Service Summon by the company Serials Solutions. It aims to offer the users of the University of Konstanz Library an effective, efficient and satisfactory search for academic information, which should at the same time be easy and intuitive to use and correspond as far as possible to the search habits and preferences of the users. The usability study was performed in collaboration with the Human-Computer Interaction working group in the Faculty of Computer and Information Science at the University of Konstanz. Different usability research methods were used: a focus group and an online survey, a summative and a formative user test as well as an eye-tracking study.Students from the University of Konstanz were selected as test subjects. The results of the evaluation suggest that the new search engine has a relatively high usability. Specific problems concerning the search with KonSearch were also identified, and this report demonstrates specific recommendations for action to solve these.
List of contents
Abstract ...II List of contents ... III List of illustrations ... IV List of tables ... VI
1 Introduction ... 1
2 Libraries, Users and Search Tool ... 3
2.1
User orientation and user research in libraries ... 32.2
Search behaviour studies ... 42.3
Library catalogues and search engines ... 63 KonSearch Usability Study ... 9
3.1
Preliminary remarks ... 93.1.1 Principles of usability research ... 10
3.1.2
Context of use of KonSearch ... 143.1.3
Areas of interest and research questions ... 153.2 Sample survey and methods used in Usability Study ... 16
3.2.1
Sample survey ... 173.2.2
Asking Users ... 193.2.3
Summative User Test ... 213.2.4
Formative User Test ... 233.2.5
Eye-Tracking ... 253.3 Results of the Usability Study ... 27
3.3.1
Asking Users ... 273.3.2
Summative User Test ... 323.3.3
Formative User Test ... 363.3.4
Eye-Tracking ... 443.4 Consolidation and interpretation of results ...
594 Summary ... 70
Bibliography ... 73
Appendix ... 76
List of illustrations
Illustration 1: System Usability Scale (SUS) ... 11
Illustration 2: After Scenario Questionnaire (ASQ) ... 13
Illustration 3: Test subject recruitment notice ... 18
Illustration 4: Set up of eye-tracking study ... 26
Illustration 5: Requirement categories sorted according to importance (asking users) ... 29
Illustration 6: Comparison of fulfilment of requirements of KonSearch & OPAC (asking users) ... 30
Illustration 7: Requirements of the category „sorting of results‟ sorted according to importance (asking users) ... 31
Illustration 8: Duration of task performance& number of mouse clicks (summative user test)... 33
Illustration 9: SUS evaluation for OPAC & KonSearch (summative user test)... 36
Illustration 10: Illustration of narrowing down according to publication date (formative user test)... 38
Illustration 11: Save icon (formative user test) ... 39
Illustration 12: Location of saved entries in KonSearch (formative user test)... 39
Illustration 13: RSS feed in KonSearch (formative user test) ... 40
Illustration 14: Input problems with the search with KonSearch (formative user test)... 41
Illustration 15: Recommendations for the entry of search terms (formative user test)... 41
Illustration 16: Sort function in KonSearch (formative user test) ... 42
Illustration 17: Availability display in KonSearch (formative user test) ... 42
Illustration 18: Link resolver intermediate page (formative user test) ... 43
Illustration 19: Heat map showing the viewing fields of the KonSearch home page (eye-tracking) ... 45
Illustration 20: Fixation on viewing fields on the home page (eye tracking) ... 46
Illustration 21: Heat map showing the viewing fields of the KonSearch results list (eye-tracking) ... 47
Illustration 22: Fixation on viewing fields in the list of results
(eye-tracking) ... 48
Illustration 23: Heat map showing the viewing fields in the advanced search
(eye-tracking) ... 49 Illustration 24: Fixation on viewing fields in the advanced search
(eye-tracking) ... 50 Illustration 25: Fixation when searching for a specific title in the list of
results (eye-tracking) ... 52 Illustration 26: Duration of completion of search for a specific title in the
list of results (eye tracking) ... 53 Illustration 27: Assignment of meaning of icons ... 54 Illustration 28: Completion time of search for specific KonSearch
functions (eye-tracking) ... 55 Illustration 29: Fixation on viewing fields when refining search
(eye-tracking) ... 56 Illustration 30: Use of search filters (eye-tracking) ... 57 Illustration 31: SUS evaluation for individual test subjects
(eye-tracking) ... 58 Illustration 32: SUS evaluation for KonSearch (eye-tracking) ... 59 Illustration 33: Recommendation for improvement of the captioning of the
input field ... 61 Illustration 34: Recommendation for improvement of the highlighting of
availability ... 62 Illustration 35: Recommendation for improvement of the highlighting of the
sort function ... 62 Illustration 36: ASQ evaluation for different task types
(summative and formative user test) ... 63 Illustration 37: ASQ evaluation for the search for an eBook
(summative and formative user test) ... 64 Illustration 38: Recommendation for improvement of the structure
of the graphic for narrowing down according to publication date ... 65 Illustration 39: Recommendation for improvement of RSS feed ... 66 Illustration 40: Recommendations for improvement for the function
„preserve search settings‟ ... 66
Illustration 41: Recommendation for improvement of the saving of results ... 67
Illustration 42: SUS evaluation (summative user test and eye-tracking) ... 69
List of tables
Table 1: Verbalisation of the SUS ... 12
Table 2: Research questions in the KonSearch Usability Study ... 16
Table 3: Profile of the test subjects in the focus group ... 20
Table 4: Profile of the test subjects in the summative user test ... 22
Table 5: Profile of the test subjects in the formative user test ... 24
Table 6: Profile of the test subjects in the eye-tracking study ... 25
Table 7: Ranking of requirements in the individual categories (asking users) ... 28
Table 8: Number and seriousness of interaction problems (summative user test)... 34
Table 9: ASQ evaluation for KonSearch & OPAC (summative user test)... 35
Table 10: ASQ evaluation for selected tasks (formative user tests) ... 37
Table 11: ASQ evaluation (formative user test) ... 44
Table 12: Most important requirements of a literature search tool ... 60
Table 13: Consolidation of the ASQ evaluations
(summative user test and formative user test) ... 68
1 Introduction
Information research in academic libraries has changed drastically in recent decades: book or card catalogues are now extremely rare; today library users do not even have to use the library premises for their research. Access via the home page and the online catalogue (abbreviated to OPAC = Online Public Access Catalogue) of a library usually represents the first contact between a library and its users, when they are searching for information and media. It is therefore possible to search for information via the Internet irrespective of time and location.But as if that were not enough, another change is taking place, which is currently presenting libraries with new challenges: web search engines are beginning to rival the OPACs of academic libraries. A majority of users prefer the simple and fast search via Google etc. to the more complex catalogue searches, which means at the same time that the search behaviour of the users is influenced by the act of searching for information with web search engines.
One possible reaction of libraries is to try to change the search behaviour of users when searching for academic information. This is already an established method in the form of information literacy courses in many university libraries, although the effects on the actual search behaviour and the level of accessibility of the relevant target groups remain open to discussion. On the other hand, libraries can change the search tools that they provide for their users to suit their habits and preferences. That would in concrete terms mean adapting the search options, the results display, etc. to the current web search engines.
Thus the University of Konstanz library has, after intensive preliminary discussions in the summer of 2010, decided to purchase Summon, a web-based Resource Discovery System of the company SerialsSolutions. The aim is to connect Summon with the printed collections of the online catalogue as well as with the electronic collections of the library to create an additional new and easier to use search tool for the library users: KonSearch, the literature search engine of the University of Konstanz. The restriction to a single access point, via which all physical and electronic collections of the library can be searched, the simple refinement of search results and their sorting via relevance ranking are only some of the changes targeted by the acquisition of Summon.
One question that arose repeatedly in the preliminary stages of the implementation of KonSearch however, was how these changes would impact on the users. In order to adapt search tools in libraries to the search behaviour of thier users, it is advisable also to examine how they will cope with the functionalities of the new online search and how satisfied they are with it. For this purpose a usability study for the evaluation of KonSearch was conducted as part of library traineeship in the University of Konstanz Library. It was organised and implemented in collaboration with the Human-Computer Interaction working group in the Faculty of Computer and Information Science at the University of Konstanz.
This final report on the project aims now to summarise the background, methods and results of the study.
The first question addressed is why libraries react to changes in the search behaviour of users at all. A fundamental user orientation forms the basis for this, as well as for evaluations of old and new search tools by library users. Then the specific changes in the search behaviour of users of academic libraries are more closely determined from previous studies. The focal point in this regard concerns the consequences of these changes on the search tools provided by libraries.
Then the usability study itself is described. First of all the organisational background, principles of usability research as well as the specific questions of the study are explained in more detail.
Then the sample survey and method used in the study are described. As four different usability research methods are used for the evaluation of KonSearch, each with different key points for study, these are described in detail in succession. Then the descriptive analysis of the study is depicted separately for each individual method. Finally the results are consolidated and interpreted with regard to the underlying questions as well as possible recommendations for action.
The most important points are then summarised again in the concluding section.
At this point I would like to thank all those members of the Human-Computer Interaction working group who took part in the evaluation and in particular: Prof. Dr. Harald Reiterer, the head of the working group, Stefan Dierdorf and Svenja Leifert, the Usability Engineering:Evaluation course leaders in the 2011 summer semester, as well as those who took part in this course, Tobias Baube, Anja Fauth, Christoph Gebhardt, Torsten Hädrich, Markus Hankh, Eike Kleiner, Laura Lorenz, Sebastian Marwecki, Alexander Nagy, Thomas Ramm, Felix Schmidt, Benjamin Thiel and Johannes Zagermann, who have put a lot of time and commitment into this study.
Thank you also to the participants from the University of Konstanz Library, in particular the Catalogue Restoration working group: Birgit Fischer, Petra Hätscher, Ralf Hafner, Kerstin Keiper, Adalbert Kirchgäßner, Oliver Klingsöhr, Oliver Kohl-Frey, Isabell Leibing, Günther Rau, Anke Rautenberg, Claudia Schädler and Gudrun Schwarz.
2 Libraries, Users and Search Tools
2.1 User orientation and user research in libraries
The implementation of search engine technology for information research in academic libraries is not an end in itself, but rather can be seen as a reaction to specific changes in the library landscape. However, the registration of these changes and the resultant necessity for reorganisation cannot be taken for granted. If one considers the change in the function of libraries, which can be clearly gauged from its definition at various times, it becomes clear that libraries in the past did not always have a strong user orientation, even though this is a fundamental condition for the recognition of these environmental changes. Ewert and Umstätter describe the change in libraries as a development away from the concept of “Collection and storage place for books” (Ewert/Umstätter 1999, p. 958; translation by Atlas Translations) to that of a “[…]place which collects, arranges and makes available published information for the user from archival, economic and synoptic points of view” (ibid.; p. 966; emphasis by H.L.;
translation by Atlas Translations) This definition is still relevant today and makes the users of libraries rather than the books the centre of attention.
But the development of a fundamental user orientation in libraries can be seen over time, not only in terms of definition, but also in reality. Wilmsmeier describes two phases of intensive user research in German libraries, which were characterised by different accompanying circumstances (cf. Wilmsmeier 1999, p. 281 et seq.): The wave of user studies conducted at the end of the 1960s was influenced by the fact that the academic approach was focused strongly on the methods of empirical social research. At the same time a relatively favourable financial situation also enabled the practical implementation of these methods in the area of libraries.
Conversely, a number of reductions and cuts at the same time as the growing requirements of the „digital age‟ formed the framework for the second phase of more distinctive user research, which began at the end of the Nineties and continues to the present day. A feature of this is that management concepts in the area of private industry penetrated into the public sector and into the libraries under slogans such as „New Public Management‟ or „New Management Model‟.
The basic idea of this was for the most part, that a higher quality of service as well as an increase in efficiency should be achieved by consistently focusing on the needs of the users.
Again today, German libraries attach great importance to establishing their user-friendliness / their service provision / their focus on service and the like, not only internally, but also in order to communicate them to others. This is clear from many mission statements published on library homepages (cf. Raabe 2009, p. 16 et seq.) It must however be noted that an actual user orientation does not develop from the description in a mission statement, but only through specific actions focused on this. An important element with regard to the user orientation of
libraries is for example the search tools provided by them. In particular the OPAC of a library represents for many users the first contact interface with the library. In view of this, the tendency to conduct studies into search behaviour on the one hand and evaluation of search tools on the other is understandable.
Both must of course be observed in context, as search tools are used as an aid in the search for information. If therefore they are not suitable for achieving the goals followed by the library users in their search for information, it is unlikely that they will be used in the future.
But if a library actually performs its tasks for the user, it will be interested in determining information about their search behaviour and in designing search tools in such a way that they give satisfaction to the users in the search for academic information. How this works in reality will be described in the following sections by means of a short summary of current studies into the search behaviour of users of academic libraries as well as the consequences of these studies with regard to search tools.
2.2 Search behaviour studies
In recent years a great number of reports have been published on studies that deal with the behaviour of library users when searching for information. The following summary does not claim to be complete. It aims to present, as an example, key results of selected studies into the search behaviour of users with regard to the use of specific search tools used primarily in the search for electronic information.
The large-scale study „SteFi‟ (= studying with electronic academic information) of the Federal Ministry for Education and Research (Klatt et al. 2001) examined the use of electronic academic information in higher education. A written survey of almost 3000 students as part of this research project showed that when searching for academic information, students most frequently searched by browsing the Internet (cf. ibid., p. 10 et seq.). Web search engines in particular serve here as an information source, although the students find it hard to assess the quality of the search results obtained. “The students as users of electronic academic information obviously concentrate […] on easily accessible search engines in the Internet, instead of exhausting the range of electronic academic information relevant to them” (ibid.; p. 11).
A logfile analysis of the OPAC of Freiburg University library (Hennies/Dressler 2006) showed that most of the search requests of users are limited to a basic search strategy predetermined by the design of the system. Therefore the standard search mode is usually used and no changes are made to the pre-set search fields and operators (cf. ibid., p. 7). Most search requests are made via a single search field and consist of only one search term. A look at the headings accessed in the list of results showed that a title that does not appear in the top ten results is only very rarely selected (cf. ibid., p. 10 et seq.).
In a study by the Online Computer Library Center OCLC (Calhoun et al. 2009) the expectations of library users with regard to online catalogues as well as their actual use behaviour were examined. Focus groups and online surveys were used for this. The results of the examination (cf. ibid., p. 11 et seq.) clearly show that in addition to the search itself, the availability of and access to information is very important to the users. A direct combination of search results and full texts is for example very useful in the case of electronic media. Furthermore the users would like to arrive as easily as possible at an evaluation of the quality of the search results.
Supplementary materials such as lists of contents and summaries as well as a relevance ranking and a clear availability display can contribute to this. The most frequently used search strategy is the simple keyword search, which is mostly used for a thematic or explorative search. In this respect the users would like options to narrow down the number of results. If the search target is a very specific title, however, the advanced search is used in preference.
A study by the British Joint Information Systems Committee JISC (Wong et al. 2009) also deals with the search behaviour of students and researchers, which was examined by means of observations and in-depth interviews. It was shown that the users with greater experience in searching for academic information also prefer to use „internal‟ search tools such as library catalogues and databases. Inexperienced users on the other hand prefer to search using
„external‟ tools such as the Internet search engines Google or Yahoo (cf. ibid., p. 19 et seq.).
With regard to the evaluation of the search results the users seem to trust the external sources such as Google: “There appears to be a sense of trust and belief that external resources provided by Google and Google Books are reliable and relevant, and have quality materials” (ibid., p.
21). Google is therefore the most frequently used when searching for electronic information on the Internet (cf. ibid. p. 24). One reason given for this is that internal search tools do not always afford the simple and direct access to the located media that is ascribed to Google. In addition the users seem dissatisfied and overwhelmed by the fact that the different search tools differ greatly in their functionalities and requirements of use (cf. ibid., p. 37 et seq.).
The results of this study as well as eleven other studies were compared and consolidated in a meta study (cf. Connaway / Dickey 2010). The common features of the results (cf. ibid., p. 26 et seq.) include in particular the fact that search engines such as Google today represent the first access point in the search for information and the search behaviour of the users is characterised by the keyword search used there. However, the options for accessing specific information are increasingly deemed more relevant than the pre-set search itself, hence the authors state:
“[A]ccess is the biggest issue” (ibid., p. 29). Speed and comfort are also key criteria in the choice of search tools. The users feel that these are more likely to be found through direct electronic access than through the use of a physical library. Nevertheless, when faced with large amounts of results there is a need for search engine functionalities, which for example help to evaluate the relevance of search results: “This is both an access issue […] and a usability issue”
(ibid., p. 35). The quality of the information and thus also of the metadata still plays a large part as well.
The most recent of the studies presented here (Connaway/Dickey/Radford 2011) deals with the influence of „convenience‟ – that is to say comfort / simplicity1 – on information-seeking behaviour. By means of a secondary analysis of the data in two studies by the Institute of Museum and Library Services the authors come to the conclusion:
“Convenience is thus one of the primary criteria for making choices during the information- seeking process. Convenience includes the choice of the information source (is it readily accessible online or in print), the satisfaction with the source (does it contain the needed information and is it easy to use), and the time it will take to access and use the information source.” (ibid., p. 27 et seq.)
Consequently they recommend that libraries, their systems and user interfaces should be structured in such a way that they resemble familiar web services. Libraries should also ensure that their services are easy to access for the user and that it is easy to learn how to use them (cf.
ibid., p. 28).
The user behaviour that is demonstrated in the results of these studies sets new challenges for libraries. Users associate with web search engines a simple operation and a direct accessibility to information, which is also perceived as being of a high quality. Thus web search engines represent for the users an alternative to library search tools such as OPAC and databases and are even being used more frequently than these. This in turn means that even when using library search tools, users implement search strategies to which they are accustomed from web search engines, so that mostly only a few key words and no operators are used in a single search field.
It is therefore the case, “that the search engines […] teach the users „bad‟ search habits”
(Lewandowski 2010, p. 88; translation by Atlas Translations). The question is therefore how libraries react to these accepted facts about user behaviour, in order to continue to be able to fulfil the claim that they are performing their duties for the users.
2.3 Library catalogues and search engines
Essentially, libraries choose one of two possible strategies in their reactions to the user behaviour described (cf. Kohl-Frey 2011, p. 6 et seq.): the teaching of information literacy and the reorganisation of the library search tools used when seeking academic information. Whilst the teaching of information literacy aims to clarify problems and challenges for users in the search for academic information, and to indicate strategies to overcome them, the reorganisation
1 With regard to the information search convenience can best be translated by the expression „Informations- leichtigkeit‟ (cf. Kohl-Frey 2011).
of the search tools strives towards an approximation of the web search engines favoured by the users to facilitate the search itself. Or put another way: teaching information literacy aims to make users aware of and therefore change their search behaviour, whereas the reorganisation of the search tools aims to adapt these tools to the actual user behaviour. The focus of what follows is on the description of the latter strategy, i.e. the development of library search tools2. In this regard however, it must not be overlooked that only a combination of both strategies can be evaluated as an appropriate reaction to the results described, as the library users concerned must be taught how to use the „new generation‟ of search tools.
The library OPAC serves as a collection of as many media as possible, which the library provides for the users. The library holdings are described using metadata, which in turn is drawn up by librarians in accordance with specific rules and can be searched through. Problems are evident here in particular with regard to the constantly growing proportion of electronic items:
On the one hand (electronic) essays and eBooks are as a rule not entered in the OPAC. In order to search for these the users usually have to use additional search tools such as databases. On the other hand it is usually not possible to search any of the full texts with the OPAC, which would be useful for a thematic search. Subject headings are provided in the metadata, which aim to describe the content more clearly, but the users are often not aware of these. Instead of using them for a search, frequently only a keyword search is used, as the users are accustomed to doing when they use the prevalent web search engines.
To counteract these problems, catalogue enrichment and federated search systems are for example used in libraries (cf. Kostädt 2008, p. 102 et seq.). OPACs are enriched primarily by scanned tables of contents, which are rendered searchable by means of optical character recognition (= OCR) programmes. They also have the advantage for the user that they are directly available as a file through a link in the OPAC. Federated search systems on the other hand make it possible to simultaneously search different information sources with a single search request. So for example several catalogues, but also academic databases may be searched, in which essays and eBooks are indexed. However the response time of these systems is relatively long in contrast to the web search engines. In addition, sometimes no duplicate control is performed, so that a result is shown several times in the list of results, if it is indexed in different sources. Just as with sorting by date instead of by relevance this does not correspond with the expectations and search behaviour of the users, who usually take note only of the first results in a list.
In order to combat these problems, more and more libraries are now relying on search engine technologies (cf. ibid., p. 105 et seq.). Here the basic aim is to provide the user with a search tool that is just as easy to use as web search engines (only one search field) and provides high-
2 Köstadt 2008 gives a good summary of this.
quality search results in the shortest time. This is achieved by combining the different data in a single index prior to searching, rather than waiting until during the search of the different sources. Moreover, options are provided to include full texts in the search as well as a ranking of the results according to relevance. The outcome of a search therefore usually consists of a very large number of results, which is produced in a short time by the system. Then the results can be reduced by a refinement of the search using specific search facets (= selection criteria).
Kostädt names FAST Data Search, Lucene and Xapian as the most important search engines, on which these library proposals are based (cf. ibid., p. 105 et seq.). By means of these search engine technologies,new search tools have been developed by libraries themselves, such as in the case of BASE (Bielefeld), HEIDI (Heidelberg) or KUG (Cologne). However, libraries increasingly tend nowadays to buy finished products3 together with index, which are sometimes hosted not by the library itself but by the search engine provider. This is for example also the case with the Konstanz literature search engine.
Each individual case must be examined to see whether the measures described to adapt the library search tools to the changed expectations and behaviours of the users are actually successful. User research also plays a large part in this respect however. Very frequently, for example, evaluations of search tools by library users are used to assess their quality. In addition to the many satisfaction surveys conducted with the online catalogue or the search engine (cf.
e.g. Nienerza/Sunckel 2011 or Regensburg University library 2009), it is usability studies that can be applied here as well. One such study was conducted in the course of the introduction of the literature search engine KonSearch in the University of Konstanz library, and is reported in the following sections.
3 For a summary of the different products, see the reviews in SerialsBlenkle/Ellis/Haake 2009, Mayfield et al.
2008 and Stevenson et al. 2009 as well as the Ekins/Koster home page 2011.
3 KonSearch Usability Study
3.1 Preliminary remarks
In February 2010 the internal working group Catalogue Restoration of the University of Konstanz library came together for the first time, in order to advise about measures to reorganise the Konstanz online catalogue. The possible alternatives compared and assessed by the working group included catalogue extensions, a new federated metasearch (previously the Elektra system was used) or an additional search tool based on search engine technology.
Ultimately the decision was made a few months later in favour of the web-based Discovery Service Summon of the company Serials Solutions4, i.e. a search engine solution. Despite the difficult circumstances due to the closure at the same time of large sections of the library as a result of the discovery of asbestos within the building, the plans to implement Summon continued apace. At the end of 2010 the name of the new literature search engine for the University of Konstanz was fixed: it was christened KonSearch.
During the extensive preparatory work that preceded the launch of the beta version of KonSearch on 2 May 2011, the participants repeatedly attempted to take into account the point of view of the library users. In order to structure these attempts even more systematically, the writer of this report decided at the beginning of 2011 to make an evaluation of KonSearch by library users the subject of her training project. A usability study appeared very suitable for this purpose and the opportunity to work with the Human-Computer Interaction working group of the Faculty of Computer and Information Science at the University of Konstanz provided added impetus. On the one hand, the working group has at its disposal a special usability laboratory and has already gathered a great deal of experience in the conduct of usability studies, and on the other hand it had already worked together with the library several times on other projects.
The Usability Engineering: Evaluation course established the framework for the joint project, which was carried out in the summer semester of 2011. The course aimed in a lecture to provide the students with the basic methods of usability research, which they were then to apply practically in an accompanying exercise to implement their own usability study. The integration of KonSearch as an example meant that the students were faced with a situation akin to what they would come across later in the reality of their professional capacity. In the role of the
“client”, the library consulted the Human-Computer Interaction working group, in order to support the introduction of a new search system by means of an accompanying usability evaluation. This realistic approach and the fact that the participants themselves are users of the library and therefore had an interest in the topic, was responsible for the firm commitment that the students gave to the day. It was however important that they did not design their own study
4 Detailed information about Summon can be found on the company‟s home page:
http://www.serialsolutions.com/discovery/summon/ (as at: 25.08.2011)
completely autonomously, but took into account the theoretical principles of usability research on the one hand and the specific context of use of KonSearch on the other hand as well as the area of interest and particular specifications of the library.
3.1.1 Principles of usability research
The term usability cannot easily be translated into German, as it is a combination of two English words: „to use‟ and „ability‟. Perhaps the most accurate and most frequently used German translation is „Gebrauchstauglichkeit‟5, but even this term requires further explanation.
The expression is used in the area of human-computer interaction, in particular in software ergonomics. In this respect usability is considered to be a quality factor of a system for interaction with the user. Studies for the usability of a system therefore usually have the purpose of changing the system, so that the quality of the interaction can be subsequently optimised. The standard „9241: ergonomics of human-system-interaction‟ (previously:
„Ergonomic requirements for office work with visual display units‟) of the International Organisation for Standardisation (ISO) describes usability as the “extent to which a product can be used by specified users in a specified context of use, to achieve specified goals with effectiveness, efficiency and satisfaction” (DIN EN ISO 9241-11, p. 4; translation by Atlas Translations). This definition also served as a principle for the usability study discussed here.
When considering the usability of a system all individual marginal conditions must be taken into account accordingly, in particular users, context of use and aims of use. Effectiveness, efficiency and satisfaction are named as elements that constitute the usability of a system with regard to these specific conditions6. For a usability study it is also necessary first of all to define the respective users and context of use of the system as well as the aims of use. Then a study design must be developed on this basis, which permits the effectiveness and efficiency of the system as well as user satisfaction with regard to these special conditions to be measured.
This interpretation of the term implies that usability is a context-sensitive concept. According to statements about the individual usability of a system, a comparison of the usability of different systems is not however easily possible, if different marginal conditions are taken into account at any one time. Nevertheless tests were conducted to develop a scale for usability, which on the one hand aims to cover as completely as possible all three dimensions of usability and on the other hand can be consulted in order to compare the usability of different systems. An example of this is the System Usability Scale (SUS) (cf. Brooke 1996), which was developed as early as
5 The German Institute for Standardisation also uses this designation for example.
6 This concept of usability is however regarded critically (cf. Hassenzahl/Beu/Burmeister 2001) and partially extended, by for example the components Joy joy of Use use or hedonistic quality (cf. ibid. p. 70 et seq.) or the components learnability and measurability (cf. Ferré et al. 2001, p. 22).
1986 by the Digital Equipment Corporation and has proved to be a reliable7 usability scale (cf.
Bangor/Kortum/Miller 2008, p. 581 et seq.; Sauro/Lewis 2009, p. 1615). It uses a scale to show what the users each subjectively think about the usability of a system. The SUS consists of ten items in the form of positive and negative statements, which relate to the interaction with a system. They are formulated so generally, that the SUS can be used on every system. The practical approach is structured in such a way that after using the system, users rate their agreement or disagreement with the statements by using a five-point Likert scale8.
Illustration 1: System Usability Scale (SUS)
Items:
Strongly disagree
1 2 3 4
Strongly agree 5 1. I think that I would like to use the
system regularly.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a
technical person to be able to use the system.
5. I found the various functions in this system
were well integrated.
6. I thought there was too much inconsistency
in the system.
7. I would imagine that most people would
learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I
could get going with the system.
Source: Compiled by the author, based on Brooke 1996, p. 192.
Then the System Usability Scale is calculated (cf. Brooke 1996, p. 194), and the result is a score between 0 and 100. The greater this is, the higher the usability of the system is rated. However it must be considered that the SUS first of all represents the usability that an individual user assigns to the system. It is therefore also necessary to determine the average SUS of as large a number as possible of users of the system. As the SUS also enables a comparison of different systems, the interpretation shouldtake into account that in studies that use the SUS to determine
7 A measuring tool is referred to as „reliable‟ if it gives the same results for repeated measurements (cf. Häder 2010, p. 109). Reliability also means the dependability of a measuring tool.
8 A „Likert‟ scale is frequently used to measure the personal attitude to a specific object or fact (cf. Häder 2010, p.100 et seq.). It consists of several items in the form of statements, which relate to different aspects of the object or fact and are formulated as very positive or very negative. The survey subjects must show their agreement or disagreement with these statements using a graded scale (usually five levels, sometimes seven).
The different numerical values of the individual items are then added up to give a score, which represents the attitude of the survey subjects.
the usability of a system, an average score of 70 is generally awarded (cf. Bangor/Kortum/
Miller 2008, p. 577). Therefore the usability of a system can only be classified as above average if it is above this score. An SUS of below 50 is however interpreted as unacceptable (cf. ibid., p.
592).
In order to gain a more tangible impression of the usability of a system, Bangor/Kortum/Miller examined which SUS score corresponds to different verbal assessments (Bangor/Kortum/Miller 2009). In this respect it is clarified as well that an SUS score of over 50 can be interpreted as acceptable and over 70 as good.
Table 1: Verbalisation of the SUS Adjective: Average SUS:
Worst imaginable 12,5
Awful 20,3
Poor 35,7
OK 50,9
Good 71,4
Excellent 85,5
Best imaginable 90,9
Source: Compiled by the author, based on Bangor/Kortum/Miller 2009, p. 118.
In addition to the SUS the After Scenario Questionnaire (ASQ) was developed with the aim of making the usability of a system measurable (cf. Lewis 1991, p. 78). As is clear from the name, this questionnaire is used after the performance of a scenario-based9 user test with the system to be evaluated. It consists of three items in the form of statements about the performance of tasks using the system, which aim to cover all important dimensions of usability. Here also the survey subjects express their agreement with the statements by means of a scale, which unlike the SUS consists of seven levels.
9 Scenario refers to the best realistic description of a typical use situation
Illustration 2: After Scenario Questionnaire (ASQ)
Items:
Strongly agree
1 2 3 4 5 6
Stronlgy disagree 7
1. Overall, I am satisfied with the ease of
completing the tasks in this scenario.
2. Overall, I am satisfied with the amount of time it took to completethe tasks in this scenario.
3. Overall I am satisfied with the support
information when completing the task.
Source: Compiled by the author, based on Lewis 1991, p. 79.
Here the first statement concerns the effectiveness and the second statement the efficiency of the system. The satisfaction with the system interaction is determined with all three items. The aspect of support or help with the performance of tasks in the third statement does not correspond with the dimensions of usability in accordance with the ISO standard, but is nevertheless considered by Lewis to be an important factor of influence on the evaluation of the usability of a system (cf. ibid., p. 79). The ASQ is rated as reliable and valid and can be analysed by calculating the average of all three answers (cf. ibid., p.80 et seq.). The score determined in this way is thus between 1 and 7 and represents a scale for the usability that the system has with regard to the scenario examined. If subsequently an average for the ASQ scores of different scenarios covered by the system is established, a scale for the usability of the entire system can be calculated.
In addition, however, to the determination of a scale such as the System Usability Scale and the After Scenario Questionnaire methods10 of usability evaluation are used, which focus more closely on the inclusion of the individual framework conditions and exposure of concrete interaction problems. They can be divided into analytical and empirical methods (cf. Sarodnick/
Brau 2011, p.199 et seq.). With analytical methods the usability of a system is evaluated by experts11. These procedures, such as for example cognitive walkthrough and heuristic evaluation, have the advantage that a relatively small expenditure is necessary to perform them.
Experts also usually know, on which interaction components of the system to focus most closely. That can however also contribute to a narrowing of the perspective, so that not all problem fields may be exposed. Empirical methods on the other hand enable a closer consideration of the actual end users, who in this respect evaluate the particular system, which corresponds more closely to the usability concept described. Examples of these methods are
10 A good summary of the methods of usability research is given for example by Lazar/Feng/Hochheiser 2010, Nielsen 1999, Sarodnick/Brau 2011, Tullis/Albert 208.
11 Experts can include both specialists in both the field of usability research and the area in which the system is used, e.g. the library staff in the case of an OPAC.
user questionnaires, usability tests or eye-tracking. However they have the disadvantage that their practical implementation is relatively expensive, as for example they have to be conducted in a special usability laboratory because of the technical resources required. This is partly offset by the fact that generally only about five test subjects are enough to identify some 80% of the relevant interaction problems (cf. e.g. ibid., p.167; Virzi 1992, p.460; 462; 466).
The implementation of these methods as an integral part of usability engineering has the aim of evaluating a system, in order then to implement measures to improve its quality. In this regard the system when evaluated may only be at the prototype stage (on paper or as test version) or may already be fit for use. It makes sense, however, to remedy foreseeable difficulties in the interaction with the system as early as possible. Nevertheless repeated checks should still be made over time to ascertain whether any new problems exist, if the available resources permit it.
The KonSearch usability study was conducted when the system was still at the beta version stage. However, in order to be able to address the overall issue of the usability of KonSearch it was necessary first of all to describe the specific context in which KonSearch is to be used and then against this background formulate more specific research questions, which can be studied using usability research methods.
3.1.2 Context of use of KonSearch
ISO standard 4291-11 names as elements of the context of use the “user, tasks, equipment (hardware, software and materials) as well as the physical and organisational environments in which the product is used” (DIN EN ISO 9241-11, p.4). With regard to KonSearch these elements must now be determined more closely.
All users of the University of Konstanz library must potentially be considered to be users of KonSearch. In addition to students, teachers and researchers, this also includes local citizens.
But the group primarily targeted by KonSearch consists of the members of the University of Konstanz, which is already evident in the addition of „The literature search engine of the University of Konstanz‟ to the name. Again the students of the university are the focal points with regard to the tasks that can be performed with KonSearch. As the designation „literature search engine‟ already implies, the aim is to use KonSearch to find academically relevant information, which can then be included in the process of academic work. In section 2.2 it was described that with increasing experience in searching for academic information, the use of more specific search tools provided by the library such as academic databases is intensifying.
Inexperienced users such as students at the beginning of their studies, however, increasingly use Internet search engines such as Google, Yahoo or Bing to search for academic information.
KonSearch enables this user group to access information from most of the sources provided by the library (OPAC, databases etc.) simultaneously via one single access point, which should
indirectly increase the use of these resources by students. The equipment that is relevant in the use of KonSearch, consists of a PC with keyboard, mouse, Internet access and a browser software. The physical and organisational environments, in which the use of KonSearch takes place, are however difficult to define, as KonSearch can theoretically be used from any location.
It must however be said that the library and the computer rooms within the University of Konstanz as well as the students‟ homes represent typical environments. It is also possible that other people are often present during use, at least on the premises within the university.
3.1.3 Areas of interest and research questions
The concept of usability described corresponds very well to the questions of user perspective, which arose repeatedly during the preparations for the introduction of the literature search engine KonSearch. As usability explicitly includes the consideration of the relevant users and the context of use of a system, the concept is in harmony with the fundamental aim of obtaining information about the viewpoint of the library users with regard to the introduction of KonSearch.
With respect to the general usability concept the research question of the usability study was:
Can different requirements, which the users of the University of Konstanz library have in their literature search, be fulfilled with effectiveness, efficiency and satisfaction? Or does the system have specific faults that make it difficult to use? In this case, of course, options to remedy the faults must be demonstrated. Overall then.this was a question more of the qualitative determination of specific interaction difficulties than of the quantitative determination of scales to measure the usability of KonSearch. In order to examine this more general question using the methods of usability research, it was broken down into more specific subsidiary questions.
The library was first of all interested in determining the specific requirements and expectations that the library users generally have of a search tool used to search for academic information. In this connection it should also be seen whether KonSearch is able to fulfil these expectations. In addition the study aims to resolve the issue of whether KonSearch fulfils different information requirements. Interest here lies primarily in two types of search requests, the targeted search for a specific bibliographical reference and the explorative or thematic search12. It must also be examined whether KonSearch is suitable to perform these different search requests with effectiveness, efficiency and satisfaction. A special search request, which was prominent because of the objective of making the electronic media items easier to find with KonSearch, was the search for eBooks. The library also asked itself whether specific interaction problems occur in the literature search with KonSearch. If that is the case, recommendations to solve
12 Lewandowski assigns different information needs to both search types. Thus the targeted search for a specific bibliographical reference is based on a specific information need, but the explorative or thematic search is based on a problem-oriented information need (cf. Lewandowski 2010, p. 92 et seq.).
these problems must wherever possible be determined. Also of interest was the effect of the optical structure of KonSearch. It must be examined whether specific design elements strongly attract the attention of the library users and thus detract from other important content. In general, it should also be ascertained, how satisfied the library users are with KonSearch overall and with its individual functions and how high the usability of KonSearch can be classified.
Although the aim of KonSearch is not to replace the existing OPAC but to supplement it, it seemed useful when examining these questions to compare the two systems as well. Thus the following research questions were formulated:
Table 2: Research questions in the KonSearch usability study
1. Which requirements/expectations do the library users have of a literature search engine?
Does KonSearch fulfil these requirements/expectations?
2. Does KonSearch fulfil different information needs?
Are different search requests performed with effectiveness, efficiency and satisfaction with KonSearch?
3. Can electronic media items, in particular eBooks, be searched and found satisfactorily with KonSearch?
What is the usability of KonSearch with regard to this special search?
4. Which interaction problems arise when searching with KonSearch?
How can they be solved?
5. What is the effect of the design of the search interface?
Are there elements that very strongly attract the attention of the users?
6. How high is the usability of KonSearch overall?
How high is it with regard to individual functions?
7. To what extent can KonSearch be distinguished from the online catalogue that already exists with regard to these questions?
Source: Compiled by the author
The areas of interest of the library articulated with these questions also had effects on the methods used in the study, as not every research question can be answeredby using a particular method of usability research. Therefore, after the description of the sample survey, the next section presenting the study methods also explains which questions were addressed, and with which method they were addressed.
3.2 Sample survey and methods used in usability study
In order to obtain as comprehensive a picture as possible of the usability of KonSearch as well as any problems with its application, and to examine the questions described above, several methods were used to evaluate the system. The primary aim of the study was not to make a purely quantitative statement about the usability of KonSearch, as might for example be possible by means of the SUS. It aimed rather to consider the specifics of KonSearch and expose tangible interaction difficulties with the different methods.13 Also, because the library users were the focal point of interest, empirical methods were chosen for the study (see section
13 An exception here is the online questionnaire, which actually aimed to reach as many students as possible.
However this is described separately in section 3.2.2.
3.1.1). These were 1. a focus group and an online questionnaire, which were combined in the
„Asking Users‟ method, 2. a „summative user test‟, 3. a „formative user test‟ and 4. „an eye- tracking study‟. These methods were distributed over four project groups, which each consisted of between three and five participants from the course Usability Engineering: Evaluation.
As specific requirements of the relevant test subjects were partly connected with the application of the different methods, the structure of the sample survey should first be explained in the following section. This is then followed by the presentation of the individual study methods with each of the questions examined.
3.2.1 Sample survey
The students at the University of Konstanz were selected as potential test subjects considered for the study and thus supposed to constitute the population. They make up the largest group of library users and are the primary target group which KonSearch wishes to reach. In order to recruit participants for the study and thus take the sample survey from the population, no random selection was taken, which is why the results are not strictly speaking representative of the whole student population at the University of Konstanz. On the one hand the use of a random selection would have been too expensive to be applied as part of the study that was only going to last one semester. On the other hand a random selection with regard to the aims of the study did not appear necessary, as the focal point was not to make as representative as possible a statement about the usability of KonSearch, but to expose specific interaction problems. The test subjects were therefore deliberately selected for the sample survey and this was done in several stages.
In order first of all to recruit volunteers to take part in the study, several communication routes were chosen: in addition to notices within the university (see illustration 3, p. 21) people were also made aware of the study by entries in the blog on the library home page as well as on the Facebook pages of the books divisions. To compensate for expense a payment of €8 per hour was also promised by the Human-Computer Interaction working group.
Illustration 3: Test subject recruitment notice
Source: Illustrated by the project groups
[Translation of illustration 3:
Looking for Participants! € 8 per hour Aim: Evaluation of a search engine
What do you have to do? Try it out, play about a bit, answer a few questions.
Duration: approx. 1 – 1.5 hours
€ €8 an hour
? www.soscisurvey.de/uee]
As a result a total of 78 students volunteered to take part in the study. From these about ten test subjects were then selected by means of a screening questionnaire (see appendix 1) for each of the four project groups. This procedure was necessary, as the project groups had specific requirements of the test subjects, some of which differed from group to group and some of which were the same for all groups.
For example it was important for all project groups to find relatively heterogeneous test subjects for the study. Only in this way was it possible to ensure that, despite the small number of candidates, relevant interaction problems could be identified, which may concern only one sub- group of the population and are therefore dependent on a specific variable.
Criteria by which the respective test subjects were distinguished were in this case gender, course of study and duration of study. One selection criterion that was significant for the practical performance of the study, was the mother tongue of the participants. As all questionnaires, task descriptions etc. were formulated in German and mistakes due to linguistic misunderstandings were to be ruled out, German as mother tongue was a requirement in the selection of the test subjects. The aim was to try not to use any information or computer science students as test subjects for the study, as they would not only have been users of the system but also experts.
The requirements of the project groups varied depending on the method used and the relevant area of interest. Thus the test subjects for the eye-tracking study should not if possible have had any experience with KonSearch, as one of the aims was to study the initial optical impression of the system. For the summative usability test it was important that the test persons, in addition to having experience with the existing OPAC, had also already accumulated experience in searching with KonSearch, given the aim to compare both systems according to requirements that were as identical as possible. For the formative usability test however, the ideal was for the test subjects to have as varied a level of experience as possible with regard to KonSearch, which in turn was conducive to the heterogeneity of the participants in the study. Furthermore, for the eye-tracking study it was a technical requirement that the test persons did not wear glasses, as otherwise it would not have been possible to track their gaze.
The test subjects concerned were subsequently contacted by the separate project groups. They were informed in general about the relevant study and offered a time to perform the study.
After a few negative replies, the number of test subjects was reduced to six or seven for each study method. Thus a total of 25 students took part in the study14.
3.2.2. Asking Users
One project group applied the Asking Users method. The key questions here were which requirements the students at the University of Konstanz have of a literature search system in general and how well KonSearch and the existing OPAC fulfil these expectations and for which aspects this is not the case. Thus research questions 1 and 7 were studied. By integrating the specific requirements to enable a search via all publication forms (i.e. print media and electronic media) by means of a single system and in order to be able to filter the list of results according to content type (e.g. eBook), the third research question was also addressed.
A multi-stage procedure was chosen to answer these questions using the Asking Users method.
First of all the project group itself drew up a list of possible requirements of a literature search system, which was then divided into different requirement categories: search, refinement of
14 This also does not however apply to the online questionnaire (see Fn.13), which is described in section 3.2.2.
search, sorting of search results, presentation, personalisation and notification, exchange and cooperation, help.
This categorisation then served as an introduction for the focus group introduced at the next stage. This consists of a thematic discussion in a small group of test subjects, which is usually moderated by means of a conversation thread, but otherwise is not very standardised and serves to explore the facts. In this case the aim was to determine the requirements of the students of a search system on the basis of the categories compiled. The focus group consisted of six students, who differed from each other with regard to the criteria of gender, course of study and duration of study and thus constituted a heterogeneous group:
Table 3: Profile of the test subjects in the focus group Age Gender Course of
Study
Intended Degree
Number of Semesters 21 Male Mathematical
and Financial Economics
Bachelor degree 2
21 Female Philosophy Bachelor degree 2
24 Male Literature-Art- Media
Bachelor degree 6
29 Female English, French State
examenination
10
26 Male Information Technology
Doctorate 1
25 Male Information Engineering
Masters 3
Source: Compiled by the author
The implementation of the focus group gave the test subjects first of all the opportunity to become acquainted with KonSearch and its functionalities in an exploration phase. For this phase, they were given exemplary search tasks, but it was not obligatory to perform them.
During the exploration the test persons were asked to make notes about what they like, what worries them or what they do not understand and what they think is missing from KonSearch.
Then on the basis of these notes and in open dialogue they discussed which requirements the participants have of a literature search system and whether KonSearch or the existing OPAC fulfil these requirements. The requirements determined were then assigned to the previously developed requirement categories (see appendix 2).
On the basis of the preliminary considerations, the category plan and the requirements mentioned by the students, the final stage was to develop an online questionnaire. The purpose of this was to find out what significance is attached to the individual requirements and the
