Do decision makers prefer certain ROI metrics?

In the first set of questions, interviewees were presented with 12 different scenarios describing the ROI of a public health intervention compared with usual care. Details on the scenarios presented are available in appendix 14 of the Matrix report. Each scenario presented the same intervention using a different set of ROI metrics, thus isolating the effect of metrics on decisions (respondents were not told that it was the same intervention). The metrics included in the interviews were constructed to

represent the alternative ROI approaches identified in the literature (reported earlier).

Table 19 below summarises the metrics included in each scenario.

Table 19 Scenarios assessed through stakeholder engagement

Scenario Metrics

1 Cost per QALY gained 2 Net benefit

3 Total cost of implementation Life years saved

Cost per life year saved 4 Cost per QALY

Preventable burden of disease (QALYs) 5 Net cost savings

6 Total cost of implementation Deaths avoided

Cost per death avoided

7 Cumulative net cost savings per person (graph)

8 Cost

126 Weighted benefit score

Ratio of cost to benefit 9 Net cost savings per person

Productivity gains per person 10 Years to break even

11 Cost per QALY

Proportion of population eligible

Proportion of recipients who are disadvantaged 12 Benefit score

Cost per person (graph)

Respondents were asked: “Would you invest in the intervention based on the information provided?” Figure 11 summarises the responses (a breakdown of responses by stakeholder group is available in appendix 15 of the Matrix report). It demonstrates that, regardless of the metric employed to represent the intervention, most stakeholders said that they would not invest in the intervention based on this information alone. This is despite the fact that the intervention would be considered cost-effective based on accepted metrics – for example, it had a cost per QALY gained of only £3125, suggesting that decision makers require more information than the economic efficiency of an intervention.

Given the lack of variation in response, it is difficult to identify metrics that decision-makers prefer more than others from the evidence presented in figure 11. Rather it is likely that decision-makers require a range of information rather than a small number of preferred types of evidence. For example, the intervention was relatively warmly received when cost per QALY gained data was presented but less well received when cost per QALY gained data is supplemented with other data, such as the preventable burden of disease, population reached, or disadvantaged population reached. Thus, 5/19 said they would invest on the basis of cost/QALY but only 2/19 said they would when additional data were provided. This might suggest that these supplementary metrics are considered important by decision-makers, with their presence overriding the positive message provided by the cost per QALY metric.

127 However, given the size and heterogeneity of the sample, only very tentative

conclusions should be draw from this analysis.

Figure 11 Stakeholder decisions by ROI metric (n=19*) (source: Matrix 2011)

*Respondents were provided with the option of stating that they didn’t know whether they would invest in the intervention, which explains why the N varies between ROI metrics.

Following each question, respondents were asked “How useful is the information

128 provided?” and “What other information would you need?” A number of common themes emerged following these questions. Unsurprisingly, given the results reported in figure 11, respondents asked for more information in response to all metrics, stating that the data provided was insufficient to judge the intervention.

Other themes included:

In most scenarios where the information was not already available, respondents requests data on:

the number of people benefiting from the intervention

the population sub groups that benefited (for example different age groups)

the total cost of implementing the intervention.

Where scenarios presented metrics aggregated over a long period of time, respondents often stated that a shorter timescale, such as less than 5 years, would be more useful.

The importance of the timing of costs and benefits is supported by the positive

reaction to those metrics that either disaggregated costs and/or benefits over time, or which focused on the shorter term (for example, scenario 7 which depicted

cumulative net cost savings per person in a graph).

The two scenarios that presented the interventions using the type of weighted benefit score metrics that are produced through stakeholder workshops (scenarios 8 and 12) were considered confusing. This is not surprising, as the scores generated by this method have little meaning other than to those participating in the workshops, or if the methods used to arrive at the scores are described in detail. The fact that these methods were based on the subjective assessments of stakeholders also raised concerns about the validity of the resulting metrics.

Respondents also stated that the use of visual and graphic presentations of metrics, rather than just numbers, was useful (for example scenario 12).

A number of the metrics were regarded with suspicion or confusion due to their

‘black box’ nature (these included scenarios 2, 3, 4, 8, 9 and 11). That is, respondents thought the approaches lacked transparency in their methods or

129 content. This is, to an extent, to be expected given the limited space to provide

methodological background within the interview design.

A number of respondents identified a benchmark – another intervention against which to assess the intervention – as an important gap in the evidence necessary to make a decision. This was despite the fact that each analysis was clearly presented as having been undertaken comparing the intervention against usual care. This might point to the value or ranking when presenting ROI metrics for decision-makers.

Finally, in response to many of the metrics, respondents requested information on the cost savings generated by the intervention.