Getting the measure of diabetes: the evolution of the National Diabetes Audit
This paper was presented as the 2018 Arnold Bloom lecture at the 2018 Diabetes UK Annual Professional Conference held in London, UK
The 2018 Arnold Bloom lecture reviewed the development of the National Diabetes Audit (NDA) and what its measurements tell us about contemporary diabetes care. From seeds sown by the St Vincent Declaration in 1988 a national system has been established for benchmarking diabetes care across primary care settings, among specialist services and between commissioning areas in England and Wales. Wherever possible the sources of the data are routinely recorded electronic records; this supports data quality and minimises participation burden. Some overall national improvements have been documented but so have areas of stagnation and deterioration. Most strikingly, however, has been the revelation of large differences between services and geographies. A variety of system changes (service re-design) illustrate how appreciable improvements can be achieved. England and Wales now have a set of measurements of diabetes care that can focus change effort appropriately and support on-going improvement. Arnold Bloom’s old hospital provided a perfect example of NDA measurements that can quickly identify priorities for improvement at an individual service level.
Arnold Bloom was renowned for his ability to ask searching questions. So I presented the 2018 Arnold Bloom lecture as a series of answers to his imagined line of questioning about the National Diabetes Audit (NDA).
• How did it begin?
• What is a national audit?
• How does the NDA ‘measure’?
• What has been learnt from the NDA?
• How will NDA ‘Measurement’ improve the care of my patients?’
The NDA is actually a national measurement system for diabetes. So, why measure? Well, improvement is dependent on measurement. That is not to say that measurement causes improvement. However, improvement depends on changing systems. Without measurement, ideally benchmarked against peer services, it is often not obvious which system is most in need of improvement. Also, following a system change, it is not possible to be sure without measurement that the anticipated improvement has occurred.
In 1989 the authors of the St Vincent Declaration,1 which is the foundation of modern diabetes care in Europe, foresaw the need for measurement to underpin the changes they envisaged. With remarkable prescience they recognised that information technology, which was then in its infancy, would power the measurements. Particularly, they recognised the reliability and efficiency of using routinely recorded clinical data as the primary substrate. This was very important because measurement can be very onerous. Indeed, the burden of measurement in local ‘Clinical Audit’ has more often than not proved so exhausting that no time or energy has been left to make the changes that might improve any deficits identified.
The NDA, like other national clinical audits, is designed not only to give teams insight into their year-on-year clinical service delivery but also to provide them with peer comparisons. This ‘real world’ view is often more valuable than comparison with a randomised controlled trial derived standard. Thus, the DCCT and other clinical trials may clearly have established that for people with type 1 diabetes HbA1c =58mmol/mol (7.5%) is a desirable, clinical-outcome related standard. But even with rigorous patient selection and extra resource the clinical trials didn’t achieve this standard in every patient. What is most helpful to know is whether one’s service is doing as well as another similar service and, if not, whether the difference is greater or less than for other aspects of care, say blood pressure (BP) management or eye screening.
During the 1990s the technical foundations were developed for measuring and comparing diabetes care delivery using, exclusively, data extracted from routinely recorded electronic care records. Feasibility having been confirmed, the NDA was launched with the National Service Framework for Diabetes in 2003–4.
The approach throughout has been guided by four key principles.
1. Keep data gathering simple
• Share design and information governance (IG):
– Sound design is essential for reliable measurement. IG has become steadily more complex and demanding.
• Link routinely recorded data at individual person level:
– By extracting coded data from electronic clinical records wherever possible any extra recording and submission burdens are minimised. Unlike pay for performance measurements such as the Quality and Outcomes Framework every patient is included – no exceptions. Every data extract includes the NHS number so data from a number of sources can be linked (see Figure 1).
(Click figure for full size image)
Figure 1. Individual patient data are extracted from routine electronic records (central red outlined box). This is linked using NHS number and date of birth to other individual patient data repositories (amber boxes) such as hospital discharge data (HES/PEDW), national mortality data (ONS) and the National Paediatric Diabetes Audit. Supplementary individual patient specialty-specific data (pregnancy, foot care, inpatient diabetes harms), not routinely recorded in electronic records, are submitted via secure web forms (blue boxes) and linked similarly. There is also a snapshot audit (NaDIA) where non-identified records are submitted on paper forms from which results are scanned into a data repository
2. Reliability based on the four foundations of good data quality
• High participation:
– 2016–17: 3.19 million people with diabetes (95.3% of the diagnosd population):
T1D = 221 620; T2D = 2 721 520
T1D = 14 540; T2D = 178 325
• High data completeness:
– Most electronic patient record fields 90–100%; BMI, smoking, feet 75–85%; urine albumin:creatinine ratio (UACR) 50–65%;
– Most specialist fields 95–100%.
• Data accuracy:
– Clinical records (admin, measurements, lab results) – i.e. as good as is used for patient care;
– Clinical coding (HES, ONS) – much improved over the past 15 years due to drive from ‘payment by results’.
• Data validation:
– Range checks;
– Return to submitter review before sign-off.
3. Analytical rigour
• Report comparative measurements, case-mix adjusted where appropriate at every service level: GP, specialist, CCG (clinical commissioning group), STP (sustainability and transformation partnership), national reports.
• Provide a data repository for independent secondary analysis.
4. Minimise cost
• Opportunity cost:
– Clinical time and effort;
– Financial cost: <30p per patient with diabetes per year; <0.01% NHS cost of diabetes (>£10bn).
The result is a highly efficient, comprehensive and reliable set of measurements. By stages they have incorporated most of the key clinical domains of diabetes care – that is: core, initial and on-going care; treatment target achievement; long-term complications and death; CSII treatment; transition from paediatric to adult care; inpatient diabetes care; pre-gestational diabetic pregnancy; and diabetic foot care. Prevention of diabetes, medication use and experience of diabetes care will be added soon. Notable omissions, for which there are not yet suitable routine electronic records to which the NDA can link, include retinopathy screening and out-of-hospital hypoglycaemia.
What has been learnt so far from NDA measurements?
The important word in this section title is ‘learnt’. As advocated by Don Berwick, NDA measurements are not intended for ‘reward or punishment’.2 They are for learning about what to change and which changes have worked.
Improvements. There have been significant improvements in glucose and BP control. During 2007–17 the proportion of people with type 1 diabetes achieving the HbA1c =58mmol/mol (7.5%) target improved by almost one-third from 23% to 30% and for type 2 diabetes by one-tenth from 61% to 67%. For BP =140/80mmHg roughly one-quarter more are on target: type 1 diabetes increasing from 60% to 76%; type 2 diabetes increasing from 60% to 74%. For both type 1 and type 2 diabetes there have been year-on-year reductions in the proportions of people with chronic kidney disease stages 3–5. Additionally, for inpatients with diabetes, more have target glucose control, while inpatient hypoglycaemia, medication errors and hospital-acquired foot ulcers have all reduced.
Stagnation. But other measurements reveal lack of progress. For pregestational diabetic pregnancy there has been no improvement since the 2003 Confidential Enquiry into Maternal and Child Health in rates of stillbirth (x2 general population) and neonatal death (x4 general population) while congenital anomalies remain high. In hospitals, patients with type 1 diabetes have experienced an unchanging rate of inpatient onset diabetic ketoacidosis (1/25). Major and minor amputation rates at all ages remain unaltered over 15 years of measurement. And there has been no appreciable change in the age-related mortality rates with higher risks at all ages in people with type 1 diabetes and a greater relative risk in younger people (Figure 2).
Figure 2. The relative risk of death is calculated by comparing in each age group people with diabetes to the people in general population who do not have diabetes. The age-specific mortality rate ratios by type of diabetes and sex have not changed over the past 10 years. (Copyright © 2017 the Healthcare Quality Improvement Partnership)
Deterioration. And there has even been some deterioration. Thus, between 2012–13 and 2013–14 annual measurement of BMI, which had been steadily improving, declined from 83% to 77% in type 1 and 91% to 86% in type 2; subsequent falls have taken these to 75% and 83%, respectively. Similarly, between 2013–14 and 2014–15 annual measurement of UACR reduced from 64% to 56% in type 1 and 84% to 75% in type 2 diabetes; and, as for BMI, the downward drift has continued to 50% and 65%. In respect of long-term complications, vascular deaths continue to be about 33% higher in people with diabetes than in people without diabetes and, furthermore, for both type 1 and type 2 diabetes the age/sex adjusted additional risk of admission with myocardial infarction, heart failure or stroke is steadily and significantly increasing (2.5–3 times for type 1 and 1–1.7 times for type 2).
Learning from secondary analysis
The primary objective of the NDA is to measure against NICE specified standards3–6 for care processes, treatment targets and disease outcomes. But, of course, there is much that could be learnt from the now very large longitudinal NDA repository of observational data. Overcoming the IG barriers to conducting such analyses has proved problematic but, recently, secure legal systems have been established that should enable this valuable resource to be used to greater benefit.
Trailblazing analyses have uncovered unexpected but very significant relationships at all ages and for both type 1 and type 2 diabetes between NICE recommended annual care process completion rates and mortality. If less than half of care processes are completed over four to five years, mortality in the subsequent two years is more than twice that for people who receive all care processes and, even after adjustment for age, sex, ethnic group, social deprivation, smoking status, BMI, HbA1c, BP and cholesterol, the hazard ratio is still 1.7. Other analyses have shown relationships between consistently high HbA1c levels and risks of heart failure and death and an inverse relationship between age at diagnosis and diabetes-related excess risk of death.
Service level learning
This is the bit that clinicians often find disquieting. It is a great shame that the variation that is inevitably exposed by service level comparison or benchmarking is frequently interpreted as a personal professional criticism. It is probable that this has been fuelled by the inappropriate, unhelpful language sometimes attached to commentaries about service level measurements such as ‘ranking’, ‘league tables’, ‘bad apples’, ‘better/worse’ etc.
It is not widely enough understood that systems are the primary determinants of differences. Talented, capable, diligent people working flat out in a poor system will never achieve the performance of that of more pedestrian peers who have the good fortune to work in a well-designed system. Systems comprise: organisation (plans, relationships, governance); processes (the way things are done); resources (people, infrastructure); and leadership (guidance, support, encouragement).
What comparative measurement does is identify which of your systems are working as well, better or worse than those of your peers. Nor is it sufficiently recognised that case-mix differences rarely explain between-service differences. Sometimes, such as in the case of age or duration of diabetes, although the factor is an important determinant at individual patient level, the age and duration profiles do not differ much between services. And, as will be illustrated below, some factors that one might suspect to be influential, such as ethnicity and social deprivation, turn out to have little impact when subjected to statistical scrutiny.
Furthermore, there is insufficient appreciation that variation can be ‘normal cause’, i.e. ‘measurement noise’ that is statistically insignificant. Well-honed systems have a narrow spectrum of measurement noise and more chaotic systems a broad spectrum within which differences are nonetheless statistically non-significant. Either way, however, there will be measurements that show ‘special cause’ variation, i.e. they are statistically higher or lower than the group mean (symmetrical distributions) or median (asymmetrical distributions). The degree to which deviation from the mean/median is considered to reflect a genuine difference in system performance is a matter for judgement, but is usually set at a probability threshold of 1/100 or 1/1000, i.e. the likelihood of the result occurring by chance is less than 1/100 or 1/1000. Used constructively, such analyses can help to identify characteristics of well-functioning and poorly-functioning systems. Such observations can be translated into collective learning and hence improvement.
Examples of variation
Figure 3 shows examples of =2-fold variation found among general practices in one CCG and many specialist services throughout England and Wales.
Figure 3. Two-fold or greater variation among (A) general practices in a CCG (type 2 diabetes, blood pressure), or specialist services throughout England and Wales: (B) type 1 diabetes HbA1c during routine care; (C) HbA1c during pregnancy; and (D) diabetic foot ulcer healing. (Figures A and B: Data derived from NHS Digital Core Report 1 local reports published March 2018. Figure C: Copyright © 2017, Health and Social Care Information Centre. Figure D: Copyright © 2018, the Healthcare Quality Improvement Partnership, National Diabetes Audit)
Core diabetes care
Care processes. Care process completion rates are influenced by case-mix; age/sex, ethnicity, social deprivation, type of diabetes and duration of diabetes all have significant associations with the likelihood of the NICE annual care processes being completed. Furthermore, the distribution of these factors among their patients varies significantly between care providers. Accordingly, a statistical model can be used to classify results for each provider population into those where variation is ‘normal cause’ (as expected) and those where it is ‘special cause’ (above or below expected). Nonetheless, the extent of the variation is striking. For example, even at CCG/LHB level: type 1 diabetes HbA1c measurement ranges 68–94% (median 85%, IQR 83–87%); type 2 diabetes BMI measurement ranges 63–97% (median 87%, IQR 83–90%).
Somewhat counter-intuitively, provider level variations in treatment target achievement rates (Figure 3) are not explained by differences in case-mix. At individual patient level there are inverse associations with age, sex, ethnicity, social deprivation, type of diabetes and duration of diabetes but, although the profiles of these factors differ between providers, it turns out that other influences, which must relate to differences in systems, have a greater impact. This is illustrated in Table 1 at GP, CCG and specialist service levels for achievement of glucose and BP targets.
For children, the system changes driven by the introduction of a ‘Best Practice Tariff’ and regional paediatric diabetes networks have delivered a substantial improvement, with national median HbA1c falling from 72mmol/mol to 64mmol/mol over six years.7
(Click table for full size image)
Table 1. Illustration of how case-mix does not explain variation in HbA1c or BP target achievement rates between pairs of CCGs (type 2 diabetes), General Practices (type 2 diabetes) and Specialist Services (type 1 diabetes). The patients attending the organisations in which there are higher rates of treatment target achievement are not older, more affluent or more likely to be of UK heritage
National Diabetes Inpatient Audit
Medication errors are thought to underpin two of the most serious harms that may befall diabetes inpatients. Overall, the National Diabetes Inpatient Audit (NaDIA) has recorded a fall in the mean rate from 40% in 2011 to 31% in 2017. However, the improvement has not been uniform and there is still a large range of reported rates from 4% to over 60%. Where improvements have occurred, they have often been associated with the use of electronic care records and/or electronic prescribing. Improvements have been more evident on medical than on surgical wards.
National Pregnancy in Diabetes audit
It is well established, and the National Pregnancy in Diabetes audit (NPiD) has re-confirmed, that first trimester HbA1c is linked to higher rates of miscarriage, congenital anomaly, stillbirth and neonatal death. NPiD shows marginal associations between first trimester HbA1c and older age or shorter duration of type 1 diabetes but no associations for type 2. Such small, patient-based influences do not in any way explain the large between-service differences in the proportion of women achieving first trimester HbA1c 48mmol/mol or less: type 1 range 0–44% (median 12%, IQR 12–23%); type 2 range 0–73% (median 35%, IQR 30–45%). Similarly, for third trimester HbA1c, which is related to fetal overweight and neonatal intensive care admission, service level achievement of the 48mmol/mol target ranges from 0–82% (median 32%, IQR 27–50%) for type 1 (Figure 3C), and 43–100% (median 75%, IQR 62–85%) for type 2.
In the Eastern Region an area-wide, pre-pregnancy care improvement programme using a multi-faceted approach to change engagement with women who have diabetes and primary care providers has resulted in the median achievement of the first trimester 48mmol/mol HbA1c target, improving to 23% in type 1 diabetes and 58% in type 2 diabetes.
National Diabetes Footcare Audit
The National Diabetes Footcare Audit (NDFA) has confirmed a key component of NICE guidance, namely the importance of prompt assessment by a specialist interdisciplinary service (SIDS). Prolonged delays are associated with more severe ulceration at presentation and poorer outcomes (healing, hospital admission, revascularisation and amputation). Unsurprisingly, failure to commission a team and pathway is also associated with these poor outcomes. Nonetheless, further variation is revealed when the rates of achieving the key outcomes (alive and ulcer free at 12 and 24 weeks after first assessment by the SIDS: severe ulcers 12-week range 13–48% (median 33%, IQR 28–40%); 24-week range 33–77% (median 56%, IQR 49–61%); (Figure 3D). Further multivariate analysis shows that these differences, and indeed the likelihood of amputation, are not related to differences in the characteristics of the patient populations and so must reflect as yet unknown differences in the ways that SIDS work.
In the South West region work, a whole systems approach to improving footcare services, provoked by high regional amputation rates, has shown that over just a few years such changes lead to dramatic reductions in amputation rates – in one locality from 1.4 to 0.3 per thousand people with diabetes per year.
How can NDA measurements help service improvement?
Knowing that variation occurs doesn’t reduce it. Arnold Bloom would have wanted to know how it would help his patients at the Whittington Hospital. Benchmarking in national audits allows services easily to pick out where they may be falling behind their peers and thereby focus their improvement effort. A high-level 2018 Whittington Hospital ‘dashboard’ nicely illustrates this (Figure 4). It is easy to see that in contrast to excellent peer comparisons for type 1 glucose control, inpatient diabetes and pre-gestational diabetic pregnancy, foot care definitely merits investigation. Also, given the high cardiovascular risk of people with type 1 diabetes, blood pressure management in these patients might be another focus.
Figure 4. Headline National Diabetes Audit measurements 2017–18 for the hospital where Arnold Bloom worked – the Whittington Hospital. Comparison is with all other participating hospitals (green = top of distribution; amber = middle of distribution; red = bottom of distribution). The results suggest that the priority for improvement should be foot care
It is easy to be bemused by numbers and to assign to them greater or less attention than they deserve. Numbers do not themselves change anything. They are signposts. And, as illustrated, they can guide intrepid travellers to better destinations and tell them whether they have arrived. Sometimes they throw up unexpected signals alerting the importance of hitherto neglected routes. Without measurements we are feeling in the dark. With them, we can navigate with some confidence, provided it is always recognised that they require careful interpretation and are viewed as a way of learning about where to go or where you have come from, rather than how to make the journey.
I cannot imagine how it would be possible sensibly to manage the complexity of delivering modern diabetes services without the guiding lights of measurement. But translating the potential of that illumination into agencies of improvement requires the will and resolve of clinical leaders. I believe it is the responsibility of the NDA to strive to make the measurements more and more accessible and easy to interpret.
When comparisons with peers are viewed dispassionately, there will always be something that stands out as a priority for improvement. Seeking a better way of working in the identified area is a tried and tested way of achieving better outcomes. I do not think it unreasonable to expect that everyone responsible for delivering diabetes care should be familiar with the NDA numbers relevant to their service and responding accordingly. In other words, the ‘model for improvement’8 should be an integral part of ‘the way we do things’. Our patients rightly expect safe and effective care that measures up to that achieved elsewhere. The NDA
9 is the oil that can keep the improvement cycle turning.
Declaration of interests
There are no conflicts of interest declared.
1. Diabetes care and research in Europe: the Saint Vincent Declaration. Diabet Med 1990;7:360.
2. Berwick DM. Quality improvement in the NHS. Presentation at the ‘Quality improvement launch: what price quality?’ London: The King’s Fund, 23 February 2016. https://www.kingsfund.org.uk/audio-video/don-berwick-quality-improvement.