INSTITUTE OF OCCUPATIONAL MEDICINE www.iom-world.org
eteam Project:
Evaluation of the User- friendliness of the tools
J Lamb, JO Crawford, A Davis, H Cowie, K Galea,
M van Tongeren
Overview
Background and aims of operational analysis
Methodology
Recruitment and participant population
Interviews
Online survey
Results
Conclusions
Recommendations
Effectiveness of tool
Effectiveness of tool
Is the tool fit for purpose and user friendly?
Is the tool fit for purpose and user friendly?
Do different people using the tool for
the same purpose get
the same answer?
Do different people using the tool for
the same purpose get
the same answer?
2 part
Operational
Analysis
Aim: Evaluation of the user-friendliness of the tools
Ineffective user-tool interfaces could negate the
utility and realism of exposure estimates
Questions to answer…..
Is the tool understandable by (and of practical value to) the users?
How good is the software?
• design of the input mask/ system crash frequency/ bugs/
interface with other software packages/ system requirements
How easily does the user translate a given use/exposure situation into the available tool input parameters?
Does the documentation meet users’ needs with regard to clarity, user-friendliness and their level of expertise?
How well do the tools and their outputs meet user
requirements for assessment of workplace exposure?
Method: telephone interviews and online survey
Interviews & online surveys designed to identify :
•
participants’ level of experience in exposure assessment &
tool usage
•
the purpose for which tool is used
•
any difficulties experienced in installing & using the tool
•
the relevance of the tool input parameters to their situation
•
whether measured exposure data had been used as a comparison with the tool outputs and
•
whether the interviewees had used any other tools
Analysis of associations between user characteristics and
their perceptions about the tools
Recruitment
Contacts given by the Advisory Board and project team:
potential participants identified from a list of interested parties
Individuals also asked to sign- up via the project website
Representative and experienced interview group selected
Interviews used to develop online survey and obtain feedback from experienced users
Online survey link sent to list of interested stakeholders
Onward distribution also made via Advisory Board:
• registered users of the ECETOC TRA and STOFFENMANAGER and industry groups (e.g.
EUROMETAUX)
Various LinkedIn Groups:
BOHS, Control Banding, AIHA
Returns
11 interviews completed (pool 28)
295 survey respondents, of whom 213 (72%) completed full question set
Results include partial completions
Wide range of exposure assessment experience
Most returns for ECETOC TRAv2, followed by STOFFENMANAGER
20
16 7 8
6 6 5 4
28
Country of origin (%)
Germany Netherlands Italy
Belgium
United Kingdom US
France Spain Other
Number of respondents by tool
151
87 61 58
50
51
ECETOC TRAv2 ECETOC TRAv3 STOFFENMANAGER MEASE
EMKG EXPO-TOOL RISKOFDERM
Accessing / downloading tool
This category of usability was most often reported as ‘easy’ or ‘very easy’ across all of the tools
Understanding the tool layout
In general, all the tool layouts were perceived to be easily
understandable by most of the respondents
Learning how to use tool
Tools considered relatively simple to learn by most respondents,
more so for MEASE and EMKG-EXPO-tool
Generating required tool output
Similar levels of ease of generating estimates noted for all tools.
More difficulty reported for TRAv2, v3 and Stoffenmanager
Returning to tool after non-use
Generally considered easy/ neutral by respondents: more
difficulty reported for ECETOC TRAv3 and Stoffenmanager
Fixing problems with the tool
More difficulty reported for this usability category than others: most for the ECETOC TRAv2, TRAv3 and
Stoffenmanager
ECETOC TRAv2
•
Most respondents used tool for REACh purposes
•
Access, layout and
generating outputs found easy
•
First time use, learning to use the tool and fixing
problems found more
difficult than other usability factors by users
•
This was unrelated to their experience or familiarity with REACh use
• Unfamiliarity with underlying concepts more difficulty in translation into inputs, in
particular for dermal exposure estimation
• Guidance found less helpful by less experienced exposure
assessors
• Fulfilled needs of most
respondents and estimates considered appropriately conservative
• Strengths: speed/ ease of use
• Limitations: conservatism &
non-inclusion of fumes/
aerosols
ECETOC TRAv3
•
Relative newness of tool fewer respondents with
experience, no interviewees
•
Most found tool easy to use in relation to access, layout and generating outputs
•
Around 33% reported more difficulty in learning how to use tool compared with other usability categories: more so for those with less tool
experience and knowledge
• Majority found translation into tools easy or neutral (total 88%)
• Guidance found helpful/ neutral by most respondents across all
experience levels
• Most respondents felt that the tool fulfilled their needs and was
appropriately conservative
• Strengths: more realistic estimates than v2; running
multiple scenarios; dermal RMMs and broad brush approach
• Limited limitations reported because of newness of tool
MEASE
Most respondents (mainly from consultancy/ industry) found tool easy to access, download and use
Unfamiliarity with concepts less likely to categorise
usability as easy/ v easy
Most respondents positive or neutral about help and
guidance
Majority reported translation of workplace situations into the tool as easy or neutral
Majority of respondents felt that the tool met their needs (62%)
Strengths: clarity of interface; instant
visualisation of changes and limited required input
parameters
Limitations: conservatism;
single substance assessments and perceived lack of clarity about underlying dataset
EMKG-EXPO-Tool
No perceived difference in usability between users with different exposure assessment experience levels
• However, significant
differences in perception of usability with varying tool experience/ understanding
Help functions found useful by majority of respondents
Most considered the tool to
suitably overestimate exposure
Translation into tools ranged from easy (43%) to difficult (20%)
• unrelated to the person’s experience, tool knowledge or purpose of use
Roughly 50/50 split on
whether the tool did/did not fulfil user requirements
Strengths: ease/speed of
use; conservatism; simplicity
Limitations: inhalation only;
imprecise estimates; lack of estimate refinement
functionality; simplicity
Stoffenmanager
Range of experience, mostly industry respondents
Majority found all usability issues easy or neutral
Around a quarter found learning how to use,
generating outputs and fixing problems difficult
Less familiarity with tool
concepts associated with more difficulty in these activities
Those with less knowledge also found tool guidance functions less useful
Around half respondents found translation into tool inputs
straightforward
64% felt tool fulfilled needs
Strengths: ease of use;
reporting/ results storage facility & doing multiple substance assessments
Limitations: non-alignment with PROC codes;
overestimation & effort required for data input
RISKOFDERM
Considered easy to access, download, understand, learn and use by all respondent types
Help functions found useful by majority of respondent types
Translation of situations into tool - predominantly neutral responses (54%)
Mixed response regarding fulfilment of requirements 44%: 37%
More “REACh” respondents (57%) reported tool as
appropriately conservative than those who used it for other purposes (18%)
Strengths: ease of use;
perception of more accurate dermal exposures than other tools; dermal route specificity
Improvement suggested:
changing interface colour/
layout & need to address a perceived over-conservatism
Conclusions
Users generally happy with the tools & supporting documentation
No major technical/ software issues reported
Some users find some tools harder than others to learn, use & fix
Translating situations into the tools and generating estimates generally straightforward- although some reports of difficulty
Tool estimates are perceived to be appropriately conservative, but some concerns that the tools are overly so
Tools seem to fulfil requirements of majority of users, however wide range (44~80%) satisfaction reported
Main issues relate to conservatism and unrealistic estimates
Recommendations
Effective use is impacted by users’ experience
•
help functions/ guidance within the tools should take into account the wide range of experience and ability of potential users
Users must understand the limitations of the tools in terms of applicability and output
•
reading the guidance/ supporting material is essential
