The “ROI” in “Team”:
Return on investment analysis framework, indicators and data for IPC and IPE
Report from the Institute on Governance
Eddy Nason
Released, May 2013
© 2011, Institute on Governance [IOG 2011-3054]
For further information, please contact:
Eddy Nason Institute on Governance
60 George Street
Ottawa, Ontario, Canada K1N 1J4 tel: 1 (613) 562-0090 fax: 1 (613) 562-0097
info@iog.ca
The Institute on Governance gratefully acknowledges the financial support of the Health Education Task Force to undertake this research. The contents of this paper are the responsibility of the authors and do not
necessarily reflect a position of the IOG, its Board of Directors, or the Health Education Task Force.
The Institute On Governance (IOG) is a Canadian, non-profit think tank that provides an independent source of knowledge, research and advice on governance issues, both in Canada and internationally.
Governance is concerned with how decisions important to a society or an organization are taken. It helps define who should have power and why, who should have voice in decision-making, and how account should be rendered.
Using core principles of sound governance – legitimacy and voice, direction, performance, accountability, and fairness – the IOG explores what good governance means in different contexts.
We analyze questions of public policy and organizational leadership, and publish articles and papers related to the principles and practices of governance. We form partnerships and knowledge networks to explore high priority issues.
Linking the conceptual and theoretical principles of governance to the world of everyday practice, we provide advice to governments, communities, business and public organizations on how to assess the quality of their governance, and how to develop programs for improvement.
You will find additional information on our activities on the IOG website at www.iog.ca
!"#$%&'$(
This paper presents the findings of a short study to develop a framework, indicators and datasets to analyse the return on investment in interprofessional education (IPE) and interprofessional care (IPC) in Canada. The framework developed builds on previous work for the Health Education Task Force that created a framework to analyse where data existed on the inputs, processes and impacts of IPC. The ROI analysis framework created here uses the same logic model of action for IPC as the previous study, but overlays it on an impact matrix designed to aid identification of input costs and impact benefits from both IPC and IPE. The input costs are partitioned according to the main costs of IPE and IPC identified in the literature and by key stakeholders. The impact benefits are partitioned according to modified “Payback Categories” – based on the Payback Framework used in previous ROI analyses of health research funding. The findings of the study are that while there is relatively good information on what is important to IPE and IPC in terms of costs and impacts, the data to collect this information widely is not generally available. However, the study does identify ways to develop some of this data, and the framework should allow anyone interested in assessing the ROI of a specific IPE or IPC approach, to identify the costs and benefits they should attempt to capture, along with ways to capture that data.
)*+',$-.+(/,00&%1(
At a time when health care costs across Canada are increasingly coming under scrutiny, an understanding of the return on investment (ROI) in any measures taken to modify the health system has become a priority for policy mobilization. One solution to a stressed Canadian health care sector has been to prioritize interprofessional care (IPC). IPC is seen as an approach that can offer improved services, better use of existing resources and ultimately health benefits tied to potential cost-savings. The F/P/T Advisory Committee on Health Delivery and Human Resources has decided to address the issue of the ROI of IPC (and its associated educational component, interprofessional education: IPE). This report provides details on the development of a framework to analyze ROI from IPC and IPE, as well as the indicators and data sources that would populate the framework.
In the world of health care ROI analyses, cost-benefit analyses have often focused on using input- output models, such as logic models, or on output categorization approaches such as the balanced scorecard. Input-output models provide valuable information on the conceptual linking of inputs to outputs. Output categorization approaches allow simple comparisons across ROI analyses due to consistent categorization of returns. Health technology assessment (HTA) provides the template for most cost-benefit analyses in health, but HTA approaches are difficult to apply to such a broad subject as IPC and IPE across the country. This means identifying a more appropriate framework to modify for analyzing ROI from Canadian IPC and IPE.
The Payback Framework, a combination of input-output and categorization frameworks, has been used to assess the ROI from health research in numerous countries around the world. In Canada, the Payback framework forms the backbone of both the Canadian Institutes of Health Research (CIHR) and the Canadian Academy of Health Sciences (CAHS) frameworks for ROI and impact assessment. With the Payback Framework providing a description of the process of IPC and IPE, and a categorization approach that will allow comparison across different approaches to IPC in different settings, it is an approach that holds merit for assessing ROI in this subject area.
The framework developed by the IOG (shown right) also builds on previous work for HETF that addressed levels of evidence of effectiveness of IPC in Canada, linking ROI to improving evidence of effectiveness and outcomes from IPC.
By overlaying the logic model of IPC and IPE process onto a matrix of costs and benefits, the framework provides the opportunity to capture indicators of costs and track them through to the relevant benefits that arise
The proposed ROI analysis framework
as a result of those costs. This is important for IPC and IPE, where the process of working together is vital to achieving optimum outcomes. This approach also allows benefits to be attributed to IPC or IPE inputs more easily (as the logic model of process can trace inputs through process to outcomes). Costs are identified on the left of the framework, in five major areas: HR, materials, technology, infrastructure and procedures. Benefits are identified on the right, and drop into modified versions of previous Payback Framework categories: Changing knowledge, capacity building, changing practice/policy, health and health sector impacts, and wider social and economic impacts.
Indicators of costs and impacts are identified in each category of the framework (large cells of the matrix). However, due to the differences between IPC and IPE (in terms of the types of costs and impacts), each large cell is split into smaller sub-cells that represent indicators of costs or benefits from either IPC alone, IPE alone, or a combination of IPC and IPE.
To populate the framework, indicators of costs and benefits were developed based on previous studies of IPC and IPE that identified their main inputs and outcomes. These indicators were supplemented by information from people working in IPC and IPE, as well as related stakeholders (such as government health or education policy makers).
Indicators of costs were conceptually simple to identify, although collecting data on the indicators is problematic using current data sets. For example, while it is simple to recommend an indicator for the cost of working interprofessionally, the data on the time spent working interprofessionally versus uniprofessionally is very difficult to trace.
Impact indicators are identified to represent only the most important impacts from IPC or IPE – with a particular focus on impacts that represent monetary or health outcomes (including health care system outcomes), since the focus of the framework is ROI. Indicators of impact for such a broad subject area are numerous, and the ones identified in this report include: better self- management of care by patients; numbers of professionals trained in IPC; guidelines around IPC approaches; greater access to care; and improved patient satisfaction/health outcomes. All the indicators are shown in Table 2 of the main report on page 31. Identifying impact data is perhaps even more problematic than identifying cost data. This is partly due to poor understanding of impacts from IPC and IPE, partly due to the difficulty of attributing outcomes such as health or wellbeing to IPC or IPE, and partly due to the nature of IPC and IPE as providing qualitative-style impacts that are difficult to measure. Some data is available however, and this is housed in a combination of national and provincial databases (CIHI, Statistics Canada, ICES, etc.) and individual organizations (such as hospitals) or studies (such as evaluations). Extrapolating from that data can provide proxies for IPC and IPE impacts, but assumptions must be clearly stated when doing so.
To take forward this work, the IOG is recommending that the framework, indicators and data sources be tested in one or more pilot projects. These pilot projects would investigate the ROI of IPC or IPE approaches and could be based on disease categories (such as diabetes), levels of care (such as primary care) or specific approaches to care delivery (such as family health teams).
Only by testing the framework and indicators will we be able to establish where cost and impact information is either missing or spuriously included. As such, pilot tests will allow a form of sensitivity analysis for the framework.
2&"3+(45(647$+7$#(
Abstract ... ii!
Executive Summary ...iii!
Table of Contents... v!
Introduction ... 1!
Defining Cost Benefit Analysis (CBA)... 1!
Defining IPC and IPE... 2!
Building a framework... 3!
Existing frameworks for ROI analysis and IPC/IPE ... 4!
Developing a ROI analysis framework from the existing approaches ... 5!
Developing indicators... 7!
What makes a good indicator? ... 7!
Developing indicators within the ROI analysis framework for IPE and IPC... 9!
Cost indicators ... 9!
Impact indicators... 15!
Combining the logic model and the impact categories ... 30!
Identifying data... 32!
Cost data (input indicator data) ... 32!
Benefits data (impacts)... 33!
Challenges and opportunities... 36!
Challenges... 36!
Opportunities ... 37! Appendices ...A1! Appendix A: Methodology ...A2! Appendix B: Key Informants...A3! Appendix C: Interview Protocol ...A4! Interviews with key informants from care, education, policy and research...A4! Interviews with key informants from data collection and storage ...A5!
28+(9:;<=(-7(92+&0=>(:+$,%7(47(-7.+#$0+7$(&7&31#-#(5%&0+?4%@A(
-7B-'&$4%#(&7B(B&$&(54%(<C6(&7B(<C)(
Interprofessional care (IPC) is an approach to delivering health care in Canada that fits with multiple federal, provincial and territorial health policies and objectives. From pan-Canadian initiatives such as the Health Human Resources Strategy1 through to provincial initiatives such as the Ontario Family Health Teams policy,2 developing IPC – and by extension interprofessional education (IPE) that will lay the foundation for IPC – is a key component in an effective, efficient, sustainable and patient-focused health care system in Canada.
It has been acknowledged that, within the current fiscal and political environment, there is a need to show evidence of the “value for money” of IPC and IPE. While evidence exists of improvements in provider satisfaction, patient satisfaction, the effective use of different professions in health care delivery, and even improvements in health outcomes for patients, there is still a level of uncertainty about the impacts (outputs and outcomes) of IPC and IPE.
Identifying these impacts in a way that allows comparisons of costs and benefits for IPC and IPE requires understanding cost-benefit analysis (CBA), delivering the evidence on costs and benefits and linking these to the realities of effective collaboration (an integral part of IPC and IPE).
<7$%4B,'$-47(
The Advisory Committee on Health Delivery and Human Resources (ACHDHR) has been charged with the task of identifying the evidence that can show where and when IPC and IPE are effective; and in particular, cost effective. To identify this evidence, ACHDHR, through the Health Education Task Force (HETF) have commissioned the Institute on Governance (IOG) to develop a return on investment (ROI) analysis framework that can underpin any future cost/benefit analysis of IPC/IPE. It is important to identify here that the aim of this specific report (and project) is purely to describe the development of the ROI framework, not to conduct the cost-benefit analysis itself.
!"#$%$%&'()*+',"%"#$+'-%./0*$*'1(,-2'
Understanding the ways in which a change in the delivery of health care can affect the input costs and monetary benefits is key to developing a good business case for any health care change. This is evidenced by the need for organizations such as NICE (National Institute for Clinical Excellence) in the UK and Canada’s own CADTH (Canadian Agency for Drugs and Technologies in Health), both of whom develop cost-benefit data on new drugs and technologies to determine their use in the health system.
However, it is important when considering a cost-benefit analysis (CBA) for IPC and IPE, that we consider exactly what is meant by CBA. A CBA can be defined as a tool that “compares alternative [intervention]s by using a generic monetary outcome (i.e., dollars)”.3 Traditional CBAs for health care (those in the Health Technology Assessment field) use information on costs and benefits gathered from clinical trials. Thus the factors influencing benefits can be accounted for when identifying the cost-benefit (or net present value). For IPC and IPE, double
1 http://www.hc-sc.gc.ca/hcs-sss/hhr-rhs/strateg/index-eng.php
2 http://www.health.gov.on.ca/transformation/fht/fht_mn.html
3 Taken from the Public Health Agency of Canada website – available at http://www.phac-aspc.gc.ca/mh- sm/pubs/evaluation/chpt2-3-eng.php (Accessed Feb 2nd 2011)
blind randomized controlled trials are practically impossible, and identifying how costs relate to benefits requires a solid understanding of the process through which benefits arise. In addition, many benefits of IPC and IPE can be difficult to monetize.
CBAs, particularly in the health field, have tended to feature relatively common characteristics.
• There is always a common unit of measurement – this is key to CBAs and is an important part of the business case for IPC and IPE. Discount rates are important to address the timescales over which benefits accrue from IPC and IPE.
• Focus on inputs and outputs – CBAs focus on formulae that investigate the inputs to an intervention (costs) and the outputs from that intervention (benefits). They do not tend to take into account the processes involved in moving from input to output (other than to assess the costs of the process – most often lumped in with input costs). For IPC and IPE this is disingenuous since it fails to account for the complexities of collaboration that can represent both a cost and an inherent benefit (and also must be performed correctly to have effective outputs).
• Benefits accrue in the health system – In traditional HTA approaches, benefits accrue mainly to health. These are often measured using Quality Adjusted Life Years (QALYs) or other measures (often measures that can be monetized). For IPC and IPE, benefits can also accrue to the health system, but these benefits may not have good QALY data (for example, more simple procedures to perform, or procedures that improve the interaction between provider and patient etc.).
• The use of “demand” in CBAs – Commonly the benefit of a specific intervention or activity is defined by the demand of consumers to purchase or use a product. For IPC and IPE this is clearly a restrictive approach as consumer demand is often not the driving force behind health care changes.
• Opportunity costs in CBAs – Opportunity costs allow a comparison of the cost-benefit from something against a different “next best” use of the costs. For IPC and IPE this is a particularly valuable tool in relating IPC and IPE to the status quo, rather than just the absence if IPC and IPE.
CBAs in the education sector tend to take into account multiple costs (e.g. the cost of educators, educational materials and infrastructure), however they do not take into account the process of education, such as the inter-professional nature of the IPE. Common benefits in education CBAs are individual earnings, and productivity for those going through education. While important for general education cost-benefit, this approach does not sit well with health care education benefits that can be external to those being educated (e.g. improved continuity of care for individuals using the health system).
Broadly speaking, CBAs form one of a number of approaches that are used to assess the return on investment (ROI) from any intervention or new product. ROI analyses as a broader term, allow for returns (impacts) from any investment to be monetized or not. Since ROI approaches are able to include non-monetized impacts, they fit better with IPE and IPC impact assessment than the limited approach of CBA (building a balance-sheet approach to ROI).4 For this reason, through this report, we refer to the framework being developed as an ROI analysis framework.
!"#$%$%&'34('.%5'346'
There is no standard definition of IPC that is used consistently across Canada and as such we have used the definition previously used in work for HETF on building the evidence base for a
4 Donaldson, C., Currie, G. and Mitton, C., Cost effectiveness analysis in health care: contraindications, BMJ, 325:
891-894, October 2002.
business case for team-based care in Canada.5 Interprofessional care (IPC) is “the provision of comprehensive health service to patients by multiple health caregivers who work collaboratively to deliver quality care within and across settings.”6 IPC involves teams that are fluid in terms of membership and are typically based on patient needs.
Interprofessional education (IPE) is also difficult to define and has been the subject of numerous definitions in different jurisdictions and studies. We have chosen to define IPE according to the definition used in the previous study for HETF. Therefore IPE is defined as: “when two or more professions learn with, from and about each other to improve collaboration and the quality of care.”7 It is worth noting that this definition does not limit IPE to pre- or post-licensure learning, nor does it limit the location in which learning happens (be it an educational institution or clinical setting).
Since data on the costs and benefits of IPC and IPE are collected in different contexts (provinces, regions, admin regimes etc.), they are often collected under different definitions of IPC and IPE. This means that although we would like to assess IPC and IPE under standard definitions, the data currently collected will not allow that. Therefore, although we have defined the two concepts, the definitions stand as an ideal for future data collection, not as a reflection of definitions used in current data collection. For the purposes of this framework development, while the differences in definition are noted, they are not fed into the developing ROI framework.
By not acknowledging the differences in definition, we are making an assumption that the differences do not have a large impact on the collection of cost and impact data. While this is likely a false assumption, under the current data conditions and the existing provincial and regional frameworks for IPC and IPE, it is not possible to take into account in the framework, how different definitions would affect the ROI findings.
It is key to the understanding of this work, that although IPE and IPC are inextricably linked to one another, we must create a hypothetical boundary between the two areas. For the sake of this work, the definitions used allow us to define any formal approach to learning (be it mentorship in practice or within a university) as IPE. Any provision of care that leads to learning for parties involved would be deemed IPC under these definitions. To counter the argument that it is impossible to create this distinction, we will attempt to develop a framework that will allow us to assess the ROI of IPC individually, IPE individually or a combination of both IPC and IPE.
D,-3B-7E(&(5%&0+?4%@(
Outside of traditional CBAs there have been a number of approaches taken to assessing the impacts of health research and health interventions. Understanding these approaches is important in discussing the ROI for IPC/IPE, since different approaches allow descriptions of the processes involved in reaching a benefit from the intervention/funding. For IPC and IPE, these processes are vital to understanding not only the benefits that arise from IPC and IPE, but also the way that costs interact. Frameworks that can take into account the costs and benefits of IPC/IPE (categorization frameworks) and the process of performing IPC/IPE (theory of action frameworks) are the ones most likely to provide effective CBAs for IPC/IPE.
5 CPRN, Toward Building a Better Business Case for Team-Based Health Care in Canada: Final Report, December 2009.
6 Interprofessional Care Steering Committee for Ontario, Interprofessional Care: A Blueprint for Action in Ontario (Toronto: Ministry of Health and Long Term Care July 2007).
7 CPRN, 2009.
Categorization frameworks: These frameworks attempt to identify ‘buckets’ in which information should be collected on outputs, outcomes and processes. Categorization frameworks are common in private sector business as tools to assess and manage organizational performance. The classic example of a categorization framework is the balanced scorecard (BSC). The BSC identifies four categories: finance; learning and growth; customers;
and, internal business processes. Information is collected on each of these areas and used to manage organizational performance (to alter the information in each category towards a stated goal). The major problem categorization frameworks face is that they do not link actions to measures well, since they do not capture processes through which change occurs in indicators.
Theory of action frameworks: These frameworks build a model of the way that interventions (be they funding or activities) move from inputs through to outcomes. The most common example is the logic model, which builds a flow diagram of inputs, processes, outputs and outcomes based on the goals of the intervention. Theory of action frameworks are commonly used for specific programs and are common in the public and not for profit sectors. The major drawback with theory of action frameworks is that they can be too specific to the intervention being assessed and therefore do not allow comparisons with similar interventions.
67$*+$%&'#8.9":)8;*'#)8'<=3'.%./0*$*'.%5'34(>346'
One framework that combines both categorization and ‘theory of action’ approaches to evaluating initiatives (particularly in health) is the Payback Framework. This framework uses a logic model of the process of the initiative and a categorization of impacts to help provide evaluation evidence for accountability for funding, advocacy for the initiative and learning for those managing the initiative.
The Payback Framework is an approach that has become almost ubiquitous in health services analysis in recent years because of its power to describe service processes and analyze service outcomes. It has been used in Europe, Asia, Australasia and the Americas to develop evaluations of impact,8 analyze portfolios of investments9 and to develop ROI assessments.10 In Canada, the Payback framework has formed the backbone of the Canadian Institute for Health Research’s internal impact assessment strategy11 and also been the basis of the Canadian Academy of Health (CAHS) framework to facilitate the analysis of ROI from all Canadian health research funding.12 The Payback framework has also recently formed the basis for a return on investment (ROI) analysis (of which CBAs are one approach) of health research funding in the UK,13 and an analysis of impact for the Irish Health Research Board.14
By basing the ROI analysis on the Payback Framework, the authors of the UK’s medical research ROI study were able to link the inputs (funding to health research) to the outputs and
8 CIHR, It's Payback Time: New International Study to Assess Impact of Heart and Stroke Research, 2007.
Available at http://www.cihr-irsc.gc.ca/e/35034.html.
9 Wooding et al., Mapping the Impact: Exploring the payback of arthritis research, RAND Europe, Cambridge: UK, 2009.
10 Canadian Academy of Health Sciences, Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, CAHS, Ottawa: ON, 2009.
11 CIHR, Developing a CIHR Framework to Measure The Impact of Health Research: Synthesis Report of Meetings February 23, 24, and May 18, 2005, CIHR, Ottawa: ON, 2005.
12 Frank and Nason, Health Research: Measuring the Social, Health and Economic Benefits. CMAJ. 180(5). 528- 534, 2009.
13 Buxton et al., Medical Research – What’s it worth? Estimating the economic benefits from medical research in the UK, Report to the UK Evaluation Forum edn. London, UK, 2008.
14 Nason et al., Health research — making an impact: the economic and social benefits of HRB funded research, Health Research Board of Ireland, 2008.
outcomes (improved health, changing healthcare costs, and economic benefits from pharmaceuticals). This allowed not only an accurate identification of the ROI, but also a way to identify the time-lag between investment and return, and a way to attribute the return to investment in UK health research only (rather than investments from other countries).
No other single framework for analyzing either impact or ROI has been so widely used and tested in the health and health services field. As such, it is prudent to make best use of the payback framework as a point of reference for this ROI analysis framework.
Based on the recommendations of HETF that led to this CBA framework project, there is a need to build on the work from CPRN that identified a framework for analysing the evidence-base for IPC and IPE. This framework is shown below (Figure 1).
This framework for analyzing the evidence base for IPC (or IPE) builds on a logic model of the process of IPC and IPE, and incorporates the logic model as one axis of a three dimensional matrix that takes into account the level of care at which IPC or IPE is aimed (primary, secondary etc…) and the level of service delivery at which the impacts from collaboration can be felt (macro, meso, micro levels). The level at which impacts are felt is important as it allows us to identify where evidence exists of changes to: the health system (macro);
health delivery organizations/institutions (such as hospitals or primary care centres - meso); or individual practitioners/educators (micro).
!"?"/)@$%&' .' <=3' .%./0*$*'
#8.9":)8;' #8)9' +A"' "7$*+$%&' .@@8).BA"*'
Turning the existing evidence-level framework described above into a ROI analysis framework requires modifications. While the above framework allows us to identify the types of inputs, processes and outputs we might expect from IPC and IPE, and to identify where evidence on these inputs, processes and outputs are plentiful, it does not allow us to collect impact information that would underpin any ROI analysis. However, since this framework has its base in the Payback Model, the framework can form the basis for the ROI analysis.
Modification 1: Creating impact categories for IPC and IPE – basing the potential areas of benefit (or impact) on the Payback Framework’s categories of impact:
• Changing Knowledge – e.g. coming up with new ways of working collaboratively, new approaches to IPE, Changing the knowledge of individuals in care/education
• Capacity Building – e.g. increasing the number of professionals in IPE, using technology to increase the opportunities for IPC/IPE.
• Changing Practice and Policy – e.g. More collaborative ways of working, changes to educational curricula.
Figure 1. Framework for analyzing the evidence around IPC (CPRN 2009)
• Health and Health Sector Impacts – Improved health of patients (including improving the patient experience), changing costs of health care provision.
• Social and Economic Impacts – e.g. spillover effects to public health behaviour, cost savings to business through improved health outcomes.
Modification 2: Identifying costs using indicators of key inputs to IPE and IPC. This must take into account the major costs identified in the literature and by key stakeholders. It is important to consider both information sources, since costs associated with IPE or IPC environments, such as administrative costs, can often be overlooked in academic analyses.
Modification 3: Simplifying the 2nd and 3rd dimensions of the evidence framework. Since the framework for evidence of the effects of IPC and IPE includes dimensions for micro/meso/macro level impacts, and those at different levels of care, any use of additional categories of impact will create a prohibitively complex model that cannot be simply populated. While it is important to be able to assign impacts to different levels of care (showing where benefits occur), this can be added to the framework after initial development.
Modification 4: Identifying whether costs and benefits relate to IPE only, IPC only or a combination of IPE and IPC. This is important since the costs and benefits from IPC and IPE can be in very different places. For example, the costs of providing space for IPC and IPE may both require specific infrastructure, but only IPE has costs associated with linking the spaces for education and care (such as the administrative cost of integrating clinical placements with IPC environments). By separating IPC, IPE and combined IPC/IPE costs/benefits intellectually, the framework will be flexible enough to evaluate care or education approaches that have a combination of ‘IPC or IPE only’ costs/benefits and ‘IPC and IPE’ costs/benefits.
These modifications take into account the different aspects that need be addressed when considering the ROI from collaborative approaches. The framework differs from previous work on collaboration in health care, as it attempts to link costs to impacts through the process of IPC or IPE – something previous ROI analyses around health care collaboration have not done. For example, work by Coleman on hypothetical health maintenance organizations, shows how inputs and outputs from collaborative systems can be measured and analyzed, but does not include any information on the vital processes around collaboration itself.15
15 Coleman, D., Developing an ROI for Collaboration Projects or Programs, Collaborative Strategies LLC White Paper, No date.
Figure 2. ROI analysis framework for IPE and IPC
The framework shown above outlines the main categories of costs and the modified Payback Framework categories of impacts (used recently in ROI analyses for the value of UK health research funding).16 These categories of cost and benefit (input and impact) are vital to populating the frameworks matrix with indicators of input and impact that we can use to build an ROI analysis of IPE and/or IPC. The next section discusses these indicators and outlines how we have populated the matrix.
F+.+34G-7E(-7B-'&$4%#(
A framework without indicators is attractive but powerless. While the ROI analysis framework can describe the way that costs and benefits accrue to IPC and IPE, it cannot alone identify what those costs and benefits are. To do that, we need to populate the matrix with indicators of costs and impacts17
CA.+'9.;"*'.'&))5'$%5$B.+)8D'
When selecting indicators for any analysis or evaluation, there are a number of criteria that need to be taken into account. Most commonly amongst policy relevant evaluations and analyses are
16 Buxton et al., Medical Research – What’s it worth? Estimating the economic benefits from medical research in the UK, Report to the UK Evaluation Forum edn. London, UK, 2008.
17 We have very consciously used the term “impacts” and not “benefits”. Simply using the word “benefits” can often ignore “negative benefits”, where IPC or IPE may produce worse results than uniprofessional care. For a ROI analysis, it is important to be able to take negative benefits into account.
SMART indicator criteria.18 SMART indicators are Specific, Measurable, Achievable, Relevant and Timely.
• Specific – Relate to specific issues and information that can be compared over time.
• Measurable – Can be quantified or measured in some way that can be compared over time.
• Achievable – Indicators can’t relate to goals that the intervention or project being analyzed cannot achieve.
• Relevant – Relate to the goals and objectives of the intervention/project being analyzed.
• Timely – Can be collected regularly and provide information in a time frame that can be used to manage the intervention/project being analyzed.
While these criteria are fairly standard for good indicators, there are additional recent developments of attractiveness and feasibility that build on the SMART criteria. In recent work by the Canadian Academy of Health Sciences as part of their major assessment of Canadian health research impact, the international expert steering committee developed criteria along two axes of desirability for indicators.19 Attractive indicators are ones that would provide the best information for management of a project. Feasible indicators are ones that are the most practical to implement in any evaluation or analysis.
!""#$%"&'()(**+
• Validity – does it relate directly to a critical aspect of the program under study?
• Behavioural impact – does it drive behaviour in a positive direction? Is it likely to result in any negative unintended consequences?
• Simplicity – is the methodology, and the strengths and weaknesses relating to the indicator, readily apparent?
• Coverage – does it cover a large proportion of output for the fields of research to be assessed?
• Recency – does the data relate to current research performance, or does it have a more historical perspective?
• Methodological soundness – is the calculation of the indicator methodological sound and statistically robust?
• Comparable – can it be compared to other data to allow us to determine if it is a “good’
outcome?
,($*&-&.&"/++
• Data availability – do the data needed for deriving indicators exist, and do both the analysts, and those being assessed, have access to it?
• Cost of data – how expensive is it to purchase the data on license?
• Compliance costs– how labour intensive is it to extract/obtain the data?
• Transparency – can the calculations be replicated by interested parties?
• Timeliness – can the data be obtained/provided relatively quickly?
• Attribution – can the data be discretely ascribed to the unit being assessed?
18 HM Treasury, Cabinet Office, National Audit Office, Audit Commission, and Office for National Statistics Choosing the right Fabric: A framework for performance information, 2001 (available at
http://www.hmtreasury.gov.uk/media/EDE/5E/229.pdf)
Chevalier and Buckles, SAS2: A Guide to Collaborative Inquiry and Social Engagement - Monitoring and
Evaluation, The International Development Research Centre, 2008. Available at http://www.idrc.ca/en/ev-132945- 201-1-DO_TOPIC.html
19 Canadian Academy of Health Sciences, Making an Impact A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, November 2008
• Credible – does the indicator provide scope for special interest groups or individuals to game the system?
• Appropriately weighted – does the indicator carry the weight of the impact it measures?
It is most unlikely that any single indicator will fulfill all of the attractiveness and feasibility criteria. However, by developing a suite of indicators, it is possible to ensure that most (if not all) attractiveness and feasibility criteria are addressed.
!"?"/)@$%&'$%5$B.+)8*':$+A$%'+A"'<=3'.%./0*$*'#8.9":)8;'#)8'346'.%5'34('
To populate the matrix portion of the ROI analysis framework, we combined information on the key factors for providing IPC and IPE (to access input information) and the major impacts identified from IPE and IPC. For indicators to be used effectively, they must typically be “owned”
by the organization or group being evaluated. Therefore, creating indicators with the key stakeholders in IPE and IPC forms an important part of identifying appropriate measures of cost and benefit. Indicator information came from document and literature review, key informant interviews with stakeholders in the IPE and IPC process, and feedback from HETF. Indicators of cost and impacts were identified and then assessed against the feasibility and attractiveness criteria to determine their appropriateness for this study. In addition, interviewees were specifically asked to identify the most important costs and benefits they saw for IPC and IPE.
This helps to prioritize the indicators and ensure that we do not spend time and resources on gaining data on issues that may only be of peripheral importance.
As identified already, there is a need in the framework to intellectually separate IPE and IPC indicators. This allows us to develop more specific indicators – the
first of the SMART criteria – that relate to IPC or IPE. For the framework, this is seen in the partitioning of IPC, IPC/IPE and IPE specific sub-cells of the matrix of costs and benefits (Figure 3). It is important to note that the presence of a sub-cell in the matrix does not mean there will be a meaningful indicator for IPC, IPE or a combination of the two. If the literature and stakeholders have not
identified a particular sub-cell as important then there is no need to identify an indicator to fit it (as it would not pass our relevance criteria for choosing indicators).
64#$(-7B-'&$4%#(H<7G,$#I(
On the left-hand side of the ROI analysis framework, the matrix captures indicators of cost that relate to the inputs to IPC and IPE. By definition, costs are considered inputs to the process of delivering IPC and IPE and as such, cost indicators will only fall under the inputs part of the logic model (even if they relate to changing processes, they are the inputs that affect the process). These costs represent a combination of capital and recurrent costs, so it is important when defining the specific data that underpins the cost that it relates to the cost over a specific time frame. It is important to note that when developing indicators of cost, that costs can accumulate to multiple groups depending on what the cost is. For example, the cost of providing IPE in a university setting costs the university itself money, but providing IPE in a clinical placement setting is a cost borne by the health organization or system. While this is important to note in developing data collection strategies, it is not as important in identifying what information should be collected as an indicator of cost. This means cost data may be very attractive, but may not be very feasible.
Figure 3. Example of indicator sub-cells within the ROI analysis framework
Below we identify the important costs within each category of input, and also suggest indicators that could be used in the ROI analysis framework for IPE and IPC.
HR indicators: For those who have assessed the costs of providing IPC and IPE, one of the main costs identified has been that of the human resources involved in providing IPC and IPE.20 While this includes the obvious HR costs of providing the appropriate health professionals for an IPC team or trainers, lecturers or mentors for IPE delivery, it also includes the HR required to administrate IPC and IPE. Administration of IPC has been shown to have a considerable impact on the ability of the team to perform effectively, with Xyrichis and Lowton identifying team structures and processes including: organizational support, team meetings, clear goals and objectives, and audit as key factors in successful IPC teams.21 For IPE, administrative HR costs are also key in linking learners to IPC itself. This means that there is a cost in administering links between education and practice settings in both post-licensure learning, and clinical placements pre-licensure.
In amongst the factors identified by Xyrichis and Lowton is the issue of team meetings. For IPC and IPE this is important to monitor the cost of, as team meetings are additional time spent away from providing either care or education. Along similar lines, the cost of shared consultations in IPC is different to the cost of providing uniprofessional consultations. This is either through multiple health professionals being involved in a single consultation (cost increase) or through replacement of one health professional with another (cost decrease). Little research is available on the substitutions of one health professional to another on outcomes, but there is an acknowledgement that costs may be affected through this process.22 Xyrichis and Lowton also identify “audit” as a key factor in creating effective IPC teams, and this issue also arose during interviews with stakeholders in IPE delivery. Having an effective approach to understanding the way you provide IPC or IPE is key to improving its effectiveness and efficiency. As a rule of thumb, interventions and projects tend to have 10% of their budget committed to evaluating their outcomes and this suggests monitoring the cost of evaluating and auditing IPC and IPE is not insignificant in assessing the total cost of IPC and IPE.
Indicators: IPC only
Cost of shared consultations – time cost compared to uniprofessional cost (specific, attributable, relevant, timely: measurable?)
Costs of administration for the team – scheduling team meetings;
assigning team members; ensuring continuity of care for patient (specific, relevant, timely: attributable?, measurable?)
IPC and IPE
Cost of providing care/education – should be measured the same way as for uniprofessional care (specific, attributable, relevant, timely:
measurable?)
20 Canadian Institute for Health Information, Pan-Canadian Primary Health Care Indicators: Report 1, Volume 1, CIHI, 2006.
21 Xyrichis A and Lowton K. What Fosters or Prevents Interprofessional Teamworking in Primary and Community care? A Literature Review. Int J Nurs Stud.; 45(1):140-53. Epub Mar 2007.
22 M. Laurant, D. Reeves, et al., “Substitution of Doctors by Nurses in Primary Care,” Cochrane Database of Systematic Reviews 2 (2005). Art. No. CD001271. DOI: 10.1002/14651858.CD001271.pub2. Task Force Two.
Assessment of Health Care Delivery Models: A Physician Human Resource Strategy for Canada. 2007. Available at: http://www.physicianhr.ca/about/taskForce2-e.php. Kaen, K., Evidence from Systematic Reviews of Effects to Inform Policy-Making About Optimizing the Supply, Improving the Distribution, Increasing the Efficiency and Enhancing the Performance of Health Workers, Alliance for Health Policy and Systems Research, December 2006.
Time spent on team meetings – planning care/education approaches (specific, measurable, attributable, relevant, timely)
Time spent on administration – scheduling team meetings; assigning team members; audit/evaluation of IPC/IPE approach (specific, relevant, timely: attributable?, measurable?)
IPE only
Planning of IPE courses – time over and above planning uniprofessional courses (specific, attributable, relevant, timely: measurable?)
Administration costs for linking education and healthcare delivery – a) post-licensure; b) clinical placements (specific, attributable, relevant, timely: measurable?)
Materials indicators: In interviews with those delivering IPC and IPE it became clear that for IPC and IPE to be effective it is necessary to provide written materials that facilitate collaboration and learning. In the case of IPE, this means developing specific materials for the training of health professionals in an IPE environment, as traditional uniprofessional training materials do not provide appropriate information. In the case of IPC, materials that provide information that allows patients to be part of their own collaborative health care team. This inclusion of the patient in IPC has been shown in a small number of studies to be important for both health impacts and cost benefits. For example, Duquette et al. examined prenatal nutrition programs for high-risk pregnant women and found improvements in the outcomes of children and in as well generated costs savings of $8 for each dollar invested in the program. 23 In 2004, Oandasan et al. looked at the outcomes of patients who were involved in decision-making and found that they had more positive clinical outcomes.24 Hogg et al. recently looked at the performance of primary care models in delivering health promotion and found that the rate of health promotion was significantly higher in community health centres than in other models where providers in interdisciplinary practices saw health promotion as an integral part of primary health care.25
Indicators: IPC only
Cost of education materials for patients as part of IPC (specific, measurable, attributable, relevant, timely)
IPE only
Cost for producing IPE course materials (specific, attributable, relevant, timely: measurable?)
Technology indicators: Technology for IPC and IPE has been shown to be a key aspect in bringing health professionals together. This covers a wide range of technologies from improving IT for communication (e.g. telehealth, videoconferencing, etc.), through administrative technologies (e.g. better networking of printers in care environment to allow access for all team members), to the implementation of the electronic health/medical record. 26
23 Duquette MP, Payette H, Moutquin JM, Demmers T, Desrosiers-Choquette J. Validation of a Screening Tool to Identify the Nutritionally At-Risk Pregnancy. Journal of Obstetrics and Gynaecology Canada. 30(1):29-37. 2008.
Available at: http://www.ncbi.nlm.nih.gov/pubmed/18198065.
24 Oandasan, I., D'Amour, D., Zwarenstein, M., Barker, K., Purden, M., Beaulieu, M., Reeves, S., Nasmith, L., Bosco, C., Ginsburg, L., Tregunno, D. Interdisciplinary Education for Collaborative, Patient-Centred Practice.
Available from Health Canada. 2004.
25 Hogg W, Dahrouge S, Russell G, et al. Health Promotion Activity in Primary Care: Performance of Models and Associated Factors. Open Medicine. 3(3): 165-173. 2009. Available at: http://www.openmedicine.ca/article/view/233.
26 Curran V. Collaborative Care: Synthesis Series on Sharing Insights for the PHCTF. March 2007. Available at www.healthcanada.gc.ca; CPRN, 2009.
Electronic health records (EHRs) in particular are often cited as a way to support IPC, since the same information can be accessed by multiple health professionals without duplication of effort.
This can lead to significant cost savings – in one study providing an estimated net benefit of US
$86,000 per provider.27 While it is clearly true that the EHR is likely to support IPC, it is also clear that it will support all approaches to delivering health care. If the EHR is brought in for Canada, it will not provide a cost to just IPC, but to all approaches to delivering health care. As such, the cost will cancel out in any comparison of IPC with other approaches to delivering health care. However, since the cost of the EHR may lead to a differential benefit for IPC over uniprofessional care, we have maintained the cost in the ROI analysis framework. This is done since this framework will be used in support of a business case for IPC and it is key in the business case to show that the EHR will likely bring greater benefit to IPC than to other approaches to care. EMR’s are also referenced as costs to consider in the Manitoba Physician Integrated Network plan.28
The other factor in which IT may make a difference for IPC is in the power to link patients into their own care team. Technology has been identified as important in the future interactions of patients and their own health management,29 and monitoring the cost of providing that technology – such as patient friendly interfaces with their own medical records – will be important in determining the cost of implementing “true” IPC. This is mirrored by changes in the way that health professionals interact through technology with trainees,30 thus affecting IPE as well as IPC.
Indicators: IPC and IPE
Cost of technology required to link professionals to each other and to patients (attributable, relevant, timely: specific?, measurable?)
Cost of EHRs – important to be able to trace to differential benefit for IPC (attributable, relevant, timely: specific?, measurable?)
Infrastructure indicators: The literature and interviews suggest strongly that co-location of individuals leads to improved collaboration. In mental health, co-locating health professionals was shown to enhance collaboration. 31 Also, in The Enhancing Interdisciplinary Collaboration in Primary Health Care (EICP) Initiative, layout of the space in which teams work was shown to be important in supporting interprofessional communication.32 Interestingly, in Australia, co-location was shown in a systematic review of integration, coordination and multidisciplinary approaches in primary care to have no statistically significant impacts on health outcomes.33 This does not suggest that there are not other benefits of co-location however, such as ease of administration, improved patient experience or continuity of care. Co-location costs are considered change
27 Wang, S.J. et al., A Cost-Benefit Analysis of Electronic Medical Records in Primary Care, The American Journal of Medicine, 114: 397-403, April 2003.
28 Physician Integrated Network Initiative, Physician Integrated Network (PIN) Guide to Demonstration Site Plan Development, Government of Manitoba, February 2007.
29 CPRN, 2009.
30 This was mentioned in interviews with people working within the IPE sector.
31 Craven M and Bland R. Better Practices in Collaborative Mental Health Care: An Analysis of the Evidence Base.
Prepared for the Canadian Collaborative Mental Health Initiative. March 2006.
http://www.ccmhi.ca/en/products/documents/EN_BetterPracticesAbstract.pdf.
32 EICP. Interdisciplinary Primary Health Care: Finding the Answers – A Case Study Report. 2006. www.eicp.ca.
33 Tieman et al., Integration, coordination and multidisciplinary approaches in primary care: a systematic investigation of the literature, Australian National University, September 2006.
management costs in the Manitoba Physician Integrated Network plan so are also valued in existing analyses of IPC costs.34
Indicators: IPC and IPE
Cost of co-locating team members (specific, relevant, timely:
attributable?, measurable?) IPE only
Cost of housing interprofessional learners (specific, relevant, timely:
attributable?, measurable?)
Procedure indicators: For delivery of health care there are often costs associated with externally performed procedures that would not be covered by the cost of health human resources or infrastructure. For example, the cost of laboratory tests that are outsourced from the health system. In our assessment of the existing information on IPE and IPC there are no obvious procedure indicators that link specifically to IPE or IPC and as such we have not included any in the framework. This may change should specific procedures be deemed an important cost to assess for IPC or IPE – for example if IPC is shown to reduce the number of lab tests that arise for a specific condition then it would be important to capture the changing cost of those tests.
Indicators: Currently no specific procedure cost indicators identified for IPE/IPC Taking only the input section of the ROI analysis framework, we can see how the different categories of cost indicators align with particular aspects of the inputs to IPE and IPC. It is interesting to note that while funding and financing approaches are key components determining the outcome of IPE or IPC, they are not represented in the indicators of cost for IPC or IPE. This is because they represent a context in which IPC or IPE sits, rather than the measurable cost of providing IPC or IPE. This does not mean these issues aren’t important, just that in the confines of a cost-benefit or ROI analysis, they are the context of the analysis, not the subject.
34 Physician Integrated Network Initiative, Physician Integrated Network (PIN) Guide to Demonstration Site Plan Development, Government of Manitoba, February 2007.
Figure 4. Cost indicators within the ROI analysis framework for IPC and IPE
In order to use the ROI analysis framework it will be key to be able to identify all of the costs associated with the specific IPE or IPC example being analysed (for example, IPC for chronic heart disease patients). To facilitate this, the table below brings together the indicators of cost across the multiple input/cost categories. As identified earlier, not every indicator will be relevant to every ROI analysis in IPC or IPE – but the collection of input indicators should attempt to mirror the majority of costs incurred in whatever aspect of IPC or IPE is being examined.
Table 1. Input/cost indicators for IPE and IPC
HR Materials Technology Infrastructure Procedures
Input (cost)
IPC only
• Cost of shared consultations (time cost compared to uniprofessional cost)
• Costs of administration for the team (scheduling team meetings; assigning team members; ensuring continuity of care for patient)
IPC and IPE
• Cost of providing care/education (should be measured the same way as uniprofessional care)
• Time spent on team meetings (planning care/education approaches)
• Time spent on administration (scheduling team meetings; assigning team members; audit/evaluation of IPC/IPE approach)
IPE only
• Planning of IPE courses (time)
• Administration costs for linking education and healthcare delivery (a – post-licensure; b – clinical placements)
IPC
• Cost of education materials for patients on IPC IPE
• Cost for producing IPE course materials
IPC and IPE
• Cost of technology required to link
professionals to each other and to patients
• Cost of EHRs (important to trace costs to differential benefit for IPC)
IPC and IPE
• Cost of co- locating team members IPE
• Cost of housing
interprofessional learners
Currently no specific procedure costs identified for IPE/IPC
<0G&'$(-7B-'&$4%#(
Knowing the inputs to IPC or IPE is only useful if we can also identify the major impacts of IPC or IPE. Impacts here represent the sections of the logic model from process through to outcomes. This is an important issue, as the distinction between impacts and outputs and outcomes is often unclear. Impacts are the collection of actions and reactions brought about by intervention being evaluated – in this case funding IPC or IPE. Outputs are the specific actions that are directly attributable to the intervention. For IPC this would mean indicators such as the level of interaction of different health professionals in the care setting. Secondary outputs are changes to policies and practices brought about by intervention. In the case of IPE, that could mean new guidelines for IPE teaching in clinical settings. Outcomes are the ultimate goals of the intervention, the aspects of the system or society that the intervention outputs are designed to act upon. To achieve outcomes, the outputs of the intervention have to be put into action and acted upon. All of these are impacts, and it is prudent to note that impacts can also arise from the process of delivering an intervention. For example, the act of training new health professionals together may in itself bring benefits to the student population through greater satisfaction in their education experience.
Within these different levels of impact, we have identified five impact categories that represent the types of impact that IPC and IPE can have. These are based on the five categories from the Payback Framework that underpin the ROI framework and link to existing ROI approaches in health research:35
• Changing Knowledge – impacts that relate to changing knowledge in the world or the knowledge of individuals.
• Capacity Building – impacts that relate to building capacity for IPC or IPE, predominantly human capacity but also technology and infrastructure capacity.
35 Buxton et al., 2008; Nason et al., 2008.
• Changing Policy and Practice – impacts that change the written policies of an organization or decision making group, or that change the practice of individuals without a policy.
• Health and Health Sector Benefits (also Education Sector Benefits) – Impacts on the health of individuals and the working of the health sector (or education sector).
• Social and Economic Benefits – Impacts on the economy of Canada and on the population (such as public health behaviours).
The impact categories identified above represent the way that indicators can be grouped to compare across different cost-benefit analyses (since it is possible to compare capacity building impacts in primary care and secondary care as long as they relate to the inputs). They also provide a simple conceptual way of collating indicators that relate to when impacts are likely to arrive. For example, impacts in the advancing knowledge category tend to occur very close to the investment in IPC or IPE. Outcome impacts however, tend to be longer-term and occur further from the investment. This is important for a ROI analysis, as the longer-term impacts will require a discount rate for monetization, so that they relate to the input costs.36
0$11&)2+&31$%"+&)4&%$"5#*+"5+"6(+.52&%+354(.+
Knowing where to place indicator information (in impact categories) does not necessarily help in knowing where to collect indicator information in real life. For that, the easiest way to identify where to collect indicator information is to look to the process of IPC and IPE.
The logic model that forms the spine of the ROI analysis framework describes how the IPC and IPE approaches move towards having outcomes. To identify indicators for each impact category, we can use the sections of the logic model to show where impacts arise and then categorize those impacts within the different impact categories. This is the approach that has been used to develop indicators in previous Payback Framework analyses and allows a clear link between the process of the intervention being assessed (in this case IPC or IPE) and the types of impacts that arise from it. This link is important in telling the story of how impacts arise from funding and inputs to IPC and IPE – a key component in creating a business case that is compelling to decision makers.
7#5%(**+&)4&%$"5#*+
Within the logic model of IPE and IPC, the first area in which impacts can be identified is the process of performing IPE and IPC themselves. This is important to capture, as impacts from the process of IPC have already been shown to be important parts of the benefit of IPC – for example the act of IPC increases understanding of roles and responsibilities of different health professional groups.
Changing knowledge indicators: During the process of delivering IPC, health professionals must work together. This process in itself provides a learning opportunity for health professionals to change their knowledge of the role of the other health professionals in the team. Interactions between individuals help to break down the silos of health professionals, developed through cultures within health professions. 37 A better understanding of others’ roles
36 Brent, R.J. Cost benefit analysis and health care evaluations, Edward Elgar Publishing Ltd, Northampton, UK, 2003.
37 Hall P. Interprofessional Teamwork: Professional Culture as Barriers. Journal of Interprofessional Care, 19(s1):
188-196. 2005.
