Key:
NZ    Nahida Zaman
WB    Dr Wendy Bennett

NZ    Good afternoon, this is Nahida Zaman with Diabetes Radio.  My guest today is Dr Wendy Bennett.  Dr Bennett, hello and a warm welcome to Diabetes Radio.

WB    Thank you.

NZ    Dr Bennett, you are an Assistant Professor of Medicine at the Johns Hopkins University School of Medicine, Division of General Internal Medicine.  Can you please give our Diabetes Radio listeners some information on your clinical background and research focus?

WB    Clinically I am a General Internist at Johns Hopkins and my research focus is in several areas, I do comparative effectiveness research about diabetes treatments.  I also do research on obesity and women’s health.

NZ    Doctor, you recently published an important systematic review in the January 2012 issue of the annals of internal medicine.  The paper was titled Evaluation of Guideline Recommendations on Oral Medications for Type II Diabetes Mellitus.  Despite the crucial role that clinical practice guidelines have in guiding choices amongst the numerous medications available to treat type II diabetes mellitus, until your study, so little research had been conducted to ascertain about their quality and rigor.  Why has this been the case?

WB    Prior to our study there were two previous studies that identified guidelines on diabetes and they assessed the quality of the guidelines.  They actually used the same quality instrument that we use which is called the AGREE which stands for the appraisal of guidelines for research and evaluation instrument and these prior studies also compared whether the guideline recommendations were consistent with evidence but they were looking for consistency between each other so between the other guidelines and in general they reported that there was very little variation in the guideline recommendations and I would say what was unique about our study is that started off with a comprehensive evidence review and from that we identified seven evidence based conclusions and then we evaluated whether the guidelines recommendations were consistent with these conclusions as opposed to comparing the guidelines recommendations between each other.

NZ    Doctor, what prompted you to conduct this study?

WB    In 2007 we completed a comparative effectiveness review on all oral diabetes medications and our purpose was to understand whether clinical practice guidelines were using reviews like we did and incorporating the evidence from these reviews and also to rate their quality.  We felt that understanding how guideline developers used evidence reviews would also help us inform our methods and topics for future imperative effectiveness reviews to make them most applicable and translatable for guideline developers as well as for clinicians.

NZ    And what was the purpose behind your review?

WB    Our purpose was to systematically review clinical practice guidelines related to recommendations on oral medication treatment for type II diabetes and we had several purposes, one was to assess whether the guidelines were consistent with the evidence based conclusions that we had identified from the 2007 comparative effectiveness review, second was to evaluate the quality of the process that guideline developers used in creating their guidelines and finally we wanted to assess the relationship between guideline quality and consistency with the evidence based conclusions.

NZ    Dr Bennett, what data sources did you use and how did you select the guidelines you reviewed?

WB    We conducted a systematic search and we searched for clinical practice guidelines using two standard electronic databases, MEDLINE which we accessed via PubMed.  We also used CINAHL which stands for the cumulative index to nursing and allied health literature and then we used three guideline specific databases.  These were the National Guideline Clearing House which is in the United States, the National Library of Guidelines which is in the United Kingdom and the Canadian Medical Association Infobase clinical practice guidelines which is based in Canada and we conducted our search between July 2007 which was right after the initial comparative effectiveness review is published through to August 2011 and we restricted our research to these databases from the UK, Canada and the US because we thought that these three countries would be most likely to use and access the initial evidence review and so when we did our search we also applied several exclusion criteria in reviewing all of the guidelines, for example we excluded those that were not published in English that had been published prior to July 2007.  If they did not have recommendations on oral medication treatment, for example if they only had recommendations about insulin use then they were excluded and the guideline had to meet a 1990 Institute of Medicine Definition of what a guideline is which is that clinical practice guidelines were systematically developed statements to assist practitioners and about appropriate healthcare and also we excluded them if they were not sponsored by an organisation or a group so if it was a guideline by an individual then that was not considered by our definition a guideline, it was excluded.

NZ    And what were the results?

WB    The main result is that we found that not all the practice guidelines that we identified and oral medication treatment for diabetes were consistent with the current evidence and that the quality of their guideline development processes was very variable.

NZ    What conclusions did you arrive at based on these results?

WB    I think there are a couple of main messages.  We found that there was variability in the guidelines but we were able to identify a couple of guidelines that were high scoring for their quality and also had recommendations that were consistent with the 2007 evidence review and these guidelines tended to have rigorous guideline development that they use systematic searches of the literature, they graded the strength of their recommendations based on what the evidence was and then they also disclosed the conflicts of interest such as those with pharmaceutical companies that the guideline developers may have had.  We felt that clinicians may want to turn to these higher quality guidelines when they are searching for recommendations about oral medications for diabetes and the other conclusion is that because we found variability, guideline developers will need to pay more attention to their methods and potentially develop better methods to improve their rigour and quality, particularly to meet some of the standards that came out recently from the Institute of Medicine’s report entitled Clinical Practice Guidelines we can trust which lays out some standards for guideline developers in creating high quality guidelines that both patients and clinicians can use and trust.

NZ    Dr Bennett, are there any implications for policymakers based on these conclusions?

WB    I do think that our findings have implications for both policymakers and healthcare organisations who are increasingly turning to publish clinical practice guidelines to improve the quality and efficiency of healthcare and they are also using them to evaluate their performance of the health system and also the individual performance of physicians and so understanding the limitations of guidelines and where guidelines can improve which we aim to highlight in this paper, I think will also inform health policy decisions around diabetes care.

NZ    Doctor, other reviews have revealed the disparities in development of guidelines.  What made your study unique?

WB    There were two things, one is that our review was very focused so other reviews looking at guidelines were on general medical literature and we were very interested because we started with this systematic review.  That was on a very focused topic.  We only identified guidelines that had recommendations on oral diabetes medications.  Our review is also unique because of the method of starting off with the systematic review and its seven evidence based conclusions and then using those to assess whether the guidelines were evidence based which prior reviews had not done this before and we thought that this was a strength especially in light of the Institute of Medicine’s recommendations that guideline developers now utilise standard procedures which really now include use of systematic reviews of the literature not just expert opinion, not just the literature review but a systematic review when they are developing and grading their recommendations.

NZ    Poor adherence to guidelines is a current and ongoing issue in diabetes.  Could the lack of consistency of guidelines with the current evidence be a factor which leads to poor clinician uptake of guidelines?

WB    I think this is not completely clear but in areas of the field where the evidence may be less clear and less strong such as with long term complications of diabetes and which medications are best to prevent some of these complications where the evidence is really not clear, I think that there may be a lot more variability in clinician practice but in areas where the evidence is really quite clear such as in glycaemic control and compared to effectiveness around intermediate outcomes, we actually found that most of the guidelines were fairly consistent with the conclusions from the evidence review and then it becomes the next step of why there may be barriers to clinician uptake and of some of these recommendations and the conclusions from the evidence review that we did and I think probably these barriers are, there are probably many of them and some of them have to do with busy clinicians as well as other reasons why clinicians are not incorporating current evidence.

NZ    Which areas showed the most variability from current evidence in the various guidelines?

WB    The majority of guidelines had multiple recommendations that we felt were consistent with the evidence based conclusions and it is just important to note that in no cases did we identify the guidelines recommendations contradicted these conclusions so we identified 11 guidelines for our review and seven of the guidelines were consistent with the evidence based conclusion that metformin would be favoured as a first line agent and so four of the guidelines did not state that and some of them just did not state what would be the first line agent and then only five of the 11 guidelines were consistent with the conclusions that all regimens produce basically the same reduction in haemoglobin A1C so they have about equal efficacy and six of the 11 guidelines were consistent with the conclusion that metformin are associated with weak maintenance.  In terms of quality we also found a lot of variability so five of the guidelines received scores less than 50% and so that is out of 100% on the rigour of development scale and mostly this was because they did not use systematic methods to search or select for evidence and describe their methods for formulating recommendations or send their guidelines out for external review after they were done with it and many of the same guidelines that scored low on the rigour of development scale that we used for quality also scored low on another scale we applied called the editorial independent scale and this was particularly because they did not record the conflicts of interest of the guideline developers.

NZ    Dr Bennett, your study identified a number of guidelines that were rigorously developed consistent with the current evidence and had low risk for bias, which guidelines were these?

WB    The two guidelines that were both evidence based and had the highest quality scores were produced by the Canadian Diabetes Association and the National Institute for Health and Clinical Excellence from the United Kingdom.

NZ    Are there any limitations to your systematic review?

WB    There are several limitations.  Our review, as I mentioned before, is very focused on guidelines that had recommendations on oral medications for diabetes and so we did not assess recommendations related to other treatments such as insulin and we did not assess other important recommendations related to diabetes such as hypertension control, etc.  Like any systematic review, it is possibly excluded guidelines.  We also just included guidelines that were in English so it is possible were non-English guidelines that were international that we did not include in this review that may have also been very high quality and we also did not assess other aspects of the guideline development process.  We assessed two aspects of quality but we did not assess whether guideline developers brought stakeholders in when they were meeting and incorporated some stakeholder input to their process.  As well as whether recommendations were actionable by clinicians and this gets to your earlier point of why clinicians may not be applying evidence based guidelines into practice and so this we did not assess in our review but certainly in future work I think this would be an important area to understand how clinicians are able to use recommendations.

NZ    Doctor, how do you see your systematic review informing future guideline development?

WB    As we talked about in March of 2010, March of last year, the Institute of Medicine released a report that was entitled Clinical Practice Guidelines We Can Trust and so they proposed in that standards for creating clinical practice guidelines which included recommendations about using a systematic review of the evidence as assessing the benefits of harms of all the care options and our review I think assesses whether the guidelines were using a systematic review and using current evidence and so given the new IOM definition of a guideline and these new guideline development standards, I think that guideline developers will need to adapt their processes to improve their compliance with the standards and hopefully eventually improve the quality of care that patients with diabetes receive.

NZ    Dr Wendy Bennett, it has been a pleasure speaking to you, thank you so much for joining Diabetes Radio today.