Since 2001, all councils with adult social services responsibilities (CASSRs) have been required to conduct surveys of users’ experiences of social services (UES). These are national surveys carried out by CASSRs and are regarded as an important part of the overall performance framework for social care, providing councils with information about how they might improve services locally and central government and regulators with information to monitor and compare the performance of councils. CASSRs are required to submit their raw data to the Information Centre for Health and Social Care (IC), and user surveys are now an established part of CASSRs’ annual data returns to central government.
Although there has been interest in conducting a survey of carer’s experiences within the social services user survey group (an advisory and agenda‐setting group for the user survey programme), the user survey programme has until this time focused on the experiences of service users rather than their carers. The publication of the Labour government’s Carers Strategy in 2008, however, proved a sufficient impetus to develop a carers’ survey
(Department of Health, 2008) As a result the first centrally mandated national carers’ survey in England was conducted by 90 CASSRs on a voluntary basis between November 2009 and February 2010.
The UESs have been used as a source of data for PIs and it was clear that one potential use of the carers’ survey data could be to populate PIs in the performance framework. A criticism of the carer’s PI, NI 135, collected for the NIS for the period 2008 to 2010, is that it focuses on the activities of local authorities and offers limited information about carers’ experiences or outcomes. In view of this criticism, the DH was keen to develop a PI for carers that better reflects carers’ experiences and outcomes, in line with the then Labour government’s policy to realign the performance framework around the objective of improving outcomes for all (Office of the Deputy Prime Minister, 2005). This objective is echoed in the recent Coalition
government proposals for a new outcomes framework (Department of Health, 2010). A PI based on the carers’ survey would therefore fit much better with the current direction for performance measurement.
4.1 Development of the carer’s experience survey questionnaire
The Personal Social Services Survey of Adult Carers in England 2009‐10 (Fox et al., 2010) (see Appendix A) was based upon the 2008 Kent Carers’ Experience Survey (CES) which was commissioned by the Department of Health and Kent County Council (KCC) (Holder et al., 2009). Both surveys were developed by researchers at the Personal Social Services Research Unit (PSSRU). To develop the Kent CES researchers ran a series of focus groups with 20 carers to identify the aspects of service quality, and outcomes that carers felt were most relevant to their lives and general well‐being. The data collected served as a basis on which to develop questions reflecting the priorities of carers. The questions were then tested for understanding with a further 25 carers using the technique of cognitive interviewing (Willis, 2005). These interviews helped to refine the question wording and ensure that the questions were expressed in language that made sense to carers and resonated with them.
The Personal Social Services Survey of Adult Carers in England 2009‐10 built upon the KCC questionnaire with additional questions to reflect the pledges made in the Labour
government’s 2008 Carers Strategy, which was published after the Kent survey had been developed (Department of Health, 2008). Specific questions about breaks from caring were added and questions to capture carers’ experience of healthcare settings and support received from employers were introduced. A further 30 cognitive interviews with carers were carried out to test the new questions for understanding, and to ensure that the questions developed were relevant to carers living outside the Kent area. The IC carried out a pilot of the national survey with a small sample of CASSRs in February 2009, which was successful (The Information Centre, 2009). On this basis of this pilot, a voluntary national survey was recommended.
4.2 Measuring performance
The Personal Social Services Survey of Adult Carers in England 2009‐10 questionnaire covers a number of different aspects of carers’ experiences of services and life more generally. This focus, together with the size of the dataset provides an opportunity to explore several different options for potential carers’ experience PIs based upon robust empirical evidence.
Any of the survey questions could be chosen as a PI, but some may be more acceptable to stakeholders than others. To help identify which aspects of carers’ experiences should be reflected in a future PI, an advisory group consisting of stakeholders from the following organisations was convened: The Standing Commission on Carers, CASSRs, third sector organisations (Carers UK and Princess Royal Trust for Carers), Care Quality Commission, Department of Health, NHS Information Centre and Personal Social Services Research Unit (PSSRU). In addition, the group invited comments on initial proposals from a wider group of stakeholders via their networks. The advisory group agreed that two potential carer
experience PIs should be developed. One to reflect a key policy aim stated in the Carers’
Strategy: whether carers are able to have a life of their own and therefore to focus on the quality of life of carers. The second should reflect local priorities and focus on access to and quality of services.
In selecting a final PI, a key question is how will it be used in the future and by whom? There are a variety of potential users of the data and a number of ways in which these same
stakeholders could make use of it. For example, government departments and regulators may use the information to monitor performance nationally and make comparisons across different localities to identify ‘beacons’ or ‘coasting’ areas. In the past the government has chosen some PIs as targets and has attached financial rewards to achievement of the target. The general public, service users and carers could also make use of the statistics to assess the effectiveness of their local services, and depending on whether the data are available at the level of the individual service, to inform their purchasing choices between providers. CASSRs and third sector organisations might use the results to gauge the success of local initiatives for carers and CASSRs may also use the data to inform their commissioning decisions.
PIs can clearly serve a variety of functions and the way data are to be used and by whom has significant consequences for the characteristics of a PI. The desirable qualities of a PI are set out in Figure 1. Many of these qualities are in conflict with each other and choosing between them may depend on which function the PI is designed to serve. For example simplicity is often in conflict with the principle that the PI should not be able to be gamed since more
PSSRU Discussion Paper 2734 11
simple measures are usually easier to game. Clearly where financial incentives are attached to targets, or organisations are rated by their performance on the PI, it is important that the PI cannot be easily gamed. However, this goal would be less important if data were solely to be used for information since the stakes associated with poorer relative or absolute performance would be less high and there would be less incentive to game the PI. The relative balance of these qualities is a matter for debate, but informed debate clearly depends on clarity over which of the uses and users are most important and recognition of the need to compromise.
Figure 1: Desirable qualities of a PI
Acceptability: the extent to which the indicator is acceptable and relevant to those being assessed. For those undertaking the assessment, several elements are necessary for a PI to be considered acceptable. The elements include; relevance to current and future policy aims, whether the PI is easy to understand, is well‐defined and easy to calculate. It can also include aspects linked to uses, such as whether it can provide data that are comparable over time and across different locations; and whether it can be attributed to the action of CASSRs, the NHS or other providers who are accountable for the outcome. Another area of concern is around ensuring that the PI cannot be gamed and does not induce perverse incentives.
Feasibility: the extent to which valid, reliable and consistent data is available for collection and reporting in a timely manner, so the data are useful.
Reliability: the indicator should have minimal measurement error, or at least be accurate enough for its purpose. In the case of survey‐based PIs, where each service user is a rater of CASSR performance, inter‐rater reliability should be high; that is responses from raters should be similar. Where responses are variable, it is important that a good level of reliability can be achieved by increasing the number of raters.
Sensitivity: the indicator should have the capacity to detect changes in the unit of measurement, in this case variations in outcomes across CASSRs (and potentially providers) and variations in outcomes within CASSRs over time.
Validity: the extent to which the indicator accurately measures what it purports to measure.
Key issues here are whether the indicator is underpinned by evidence and/or consensus, the extent to which the indicator has the ability to accurately predict outcomes, and whether the basis for scoring and combining responses is defensible.
Adapted from a document produced by Nalyni Shanmugathasan, DH and Campbell, S.M., J. Braspenning, et al (1993). Improving the quality of health care: Research methods used in developing and applying quality indicators in primary care. BMJ 326 (7393):
816‐819.
The first two of the desirable qualities – acceptability and feasibility – are primarily a matter for decision‐makers and stakeholders to debate. Empirical data can be used to inform these debates, but this can only be collected after implementation. For example evidence related to the process of collecting the data for the PI, can be used to inform feasibility, and on‐going evidence related to the behaviour of stakeholders in reaction to the PI, can be used to inform concerns about gaming. Attribution of the PI to the action of the organisation can also be informed by detailed analysis and data collection. In contrast, the latter three features – reliability, sensitivity and validity – can be assessed empirically (at least to some extent) prior to implementation and are therefore the focus of the analytical part of this report.