Antibiotic resistance is one of the greatest challenges facing human health today. We are excited to welcome Dr. Scott Evans of the Harvard T.H Chan School of Public Health (HSPH) to the blog to discuss how the novel statistical methods he is developing could help tackle this global crisis.
About Scott Evans
Dr. Scott Evans of the Harvard T.H. Chan School of Public Health is the Director of the Statistical and Data Management Center (SDMC) for the Antibacterial Resistance Leadership Group (ARLG) and a member of the Steering Committee for the Center for Biostatistics in AIDS Research (CBAR) at HSPH. His interests include the design, monitoring, analyses, and reporting of and education in clinical trials. He is the author of more than 100 peer-reviewed publications and of three textbooks on clinical trials including Fundamentals for New Clinical Trialists.
Dr. Evans is a member of the Board of Directors for the American Statistical Association (ASA) and the Society for Clinical Trials (SCT) and is a former member of the Board for the Mu Sigma Rho (the National Honorary Society for Statistics). He is a member of an FDA Advisory Committee, and the Steering Committee of the Clinical Trials Transformation Initiative (CTTI), and serves as the Chair of the Trial of the Year Committee of the SCT. He is the Editor-in-Chief of CHANCE and of Statistical Communications in Infectious Diseases (SCID).
"In the United States about two million people each year acquire serious bacterial infections that have become resistant to antibiotics, and about 23,000 people die as a result."
Cytel: Could you give us some background on the problem of antibiotic resistance?
Scott Evans (SE) Superbug outbreaks have become a regular part of the news cycle. Superbug bacteria have become resistant to our most powerful antibiotics and are one of the world's most dangerous health threats. In the United States about two million people each year acquire serious bacterial infections that have become resistant to antibiotics, and about 23,000 people die as a result. In the European Union there are about 25,000 patients who die every year of these resistant infections. In addition to undermining our ability to fight these infectious diseases, antibiotic resistance also threatens the safety and effectiveness of many medical procedures that heavily rely on effective antibiotics, such as chemotherapy for cancers, dialysis for renal failure, neonate care, intensive care, and many surgeries like organ transplantation.
Cytel: I understand that you're the statistics and data management lead for the Antibacterial Resistance Leadership Group. Could you tell me a little bit about that group and its objectives?
SE: The Antibacterial Resistance Leadership Group, or what we refer to as the ARLG, is an NIH funded network whose mission is to prioritize design and to execute clinical research that would help reduce the public health threat of antibacterial resistance. The network is composed of many individuals --clinicians, statisticians and data managers and so forth. Clinician investigators are located all across the United States and now also internationally. I head up the Statistical and Data Management Center or SDMC for the ARLG. The SDMC’s statistical center is located at Harvard while the data management center resides at Duke.
Cytel: How did you become involved in this area of research?
SE: I have always been interested in infectious diseases and had worked in the HIV area for several years. As reports of more superbug outbreaks across the world occurred, I wanted to get involved and looked for opportunities. I also serve on a lot of Data Monitoring Committees (DMCs), and a few of the DMCs were investigating therapies for the treatment of superbug infections. Eventually I met a clinician named Henry "Chip" Chambers at University of California San Francisco (UCSF) through a DMC. Chip is a leading researcher in the area of resistant infections. This led to collaborations and grant opportunities. It's become a really fun ride because of the great colleagues with whom I get to work and because of the importance of this research.
"In some instances it's the clinician treating today and diagnosing tomorrow."
Cytel: So, what is different about the evaluation of antibiotics, that they require novel methods to be developed for evaluation and design?
SE: There are a couple of things. First, for many of these infections, particularly the acute infections that require therapy relatively immediately, there's a lack of rapid accurate diagnostics. In some instances it's the clinician treating today and diagnosing tomorrow. Patients enter the hospital or the ICU needing immediate care, but the laboratory work that helps to identify the offending pathogen and drug susceptibility information are lacking. Therefore we're doing a lot of work to see whether we can develop new diagnostics that might inform treatment more quickly and accurately than we're able to do now.
We also have a lot of benefit-risk challenges tied to serious diseases with morbidity and mortality implications. We also have potentially toxic therapies that can cause kidney injury for example, and affect quality of life. Then there are resistance implications. There are multidimensional effects of treatment and diagnostic decisions and we've been working on ways to synthesize these multidimensional effects in order to evaluate which therapies are optimal.
Cytel: What novel methods have you particularly explored?
SE: In clinical trials we've been developing methods for a benefit-risk assessment to assess global effects, tying together efficacy, safety, and quality of life. Typically what is done in clinical trials is that treatment effects are estimated for each outcome. Then the effects are formally or informally combined in some way. But this approach can miss important effects on patients and is inconsistent with the manner in which the outcomes are experienced by patients. Thus we “use the outcomes to analyze the patients rather than the patients to analyze the outcomes“. We believe that these methods are more pragmatic in that they're more closely connected to the experiences of the patient.
Perhaps I can illustrate this with a quick example. Suppose you treat 100 patients and 50% of those patients or 50 of those patients experience efficacy and 50% also experience toxicity. Let's suppose the efficacy and the toxicity are similar in importance. Now consider two scenarios. The 50% who experienced efficacy are the same patients that had the toxicity. In that particular scenario, there were no patients that had a net benefit perhaps indicating a worthless treatment. However if the 50% of patients who had benefit or efficacy are different than the patients that had the toxicity, then you've got a good treatment if you can find the right patients. But if you analyze one endpoint at a time and only see the 50% efficacy and 50% toxicity, then you never see the difference between these scenarios.
What we have to do is begin thinking about how we combine that information within the patient first. We're changing the “order of operations” in analysis, so that it is consistent with the manner in which the outcomes are experienced by the patients. This is telling us what's happening to patients so we can make better, more informed decisions.
"If you calculate a summary statistic of duration in the hospital, you may get a distorted view of what's going on. We really don't know how to interpret the two days in the hospital unless you know what else happened to the patient."
Another reason we've started to look at patient results is that many trials will measure an outcome that can be very difficult to interpret without other information for that same patient. For example, in antibiotic trials we measure how long the patient is in the hospital or how long the patient is in the ICU. Fewer days is interpreted as a better outcome. But if the patient dies on day two, then this also translates to fewer days in the hospital. Thus if you calculate a summary statistic of duration in the hospital, you may get a distorted view of what's going on. We really don't know how to interpret the two days in the hospital unless you know what else happened to the patient. This competing risk problem is another motivation to start looking within patient.
Yet another issue is that typically an efficacy analysis is conducted using an intent-to-treat or ITT population. Then a safety analysis is done on a safety population. Then, at the end these two analyses are combined into a benefit:risk analysis. But to whom does this benefit-risk analysis apply? Nobody knows. But if you synthesize information within patient first, this problem can go away.
TOn the diagnostic side, we are also thinking about how we can conduct more pragmatic evaluations of diagnostics. If I’m in a situation where I have two diagnostics, one with higher sensitivity and the other with higher specificity, which one do I choose? Well, it depends on several factors including the differences in sensitivities and specificities, the prevalence of disease, and the consequences of making a false positive or false negative error. We've been working on ways to tie all of those things together in a systematic way so that you can make an informed decision about which diagnostic may give you the best outcome.
Here we’ve been using the concept of “diagnostic yield” to describe the clinical or public health impact of a diagnostic application. We've developed a statistic called “weighted accuracy” that accounts for the difference in importance in the types of errors you could make in applying a diagnostic. You could have a false positive error or you could have a false negative error, but those are not the same errors. One may have more dire consequences than another and just measuring how often you're right isn't telling the full story. These methods are also able to incorporate the prevalence of disease into the calculation because this also affects diagnostic yield and the interpretation of the utility of your diagnostic.
"We have to be able to explain why a new approach produces more informative and useful results."
Cytel: What obstacles are there to implementing some of these methods?
SE: I think the biggest challenge is that new methods are a change of culture. Researchers, perhaps like everybody else, get into habits with their approaches. It can sometimes be uncomfortable to deviate from them. There's also the issue that new methods often take more work and more thought to implement than the old methods, because they're new and you have to iron out all of the wrinkles. We have the ideas, now we need to figure out how do you implement the ideas. Some people are excited by these new ideas and by the prospect of something better and more informative. But for other people it is scary and unsettling.
I think the solution is education. We have to be able to explain why a new approach produces more informative and useful results. It takes depth of understanding and a lot of practice and care to figure out how to explain new ideas in a way that somebody else can understand, perhaps someone that has a different background. You have to be able to share your ideas. This is part of progress and the evolution of thought, science, medicine, and statistics. Figuring out how to share your ideas is part of that process.
Cytel: Can these ideas be applied to other areas than antibiotics research? Do you see a space for them?
SE: Absolutely. The statistical ideas and concepts can be generalized and then tailored to new medical areas. Obviously you have to think carefully about how to tailor them to a new disease and a new treatment. We've seen early returns on this already. Many of the ideas are getting picked up in other areas. I've seen applications in neurology, cardiovascular disease, oncology and other areas of medicine.
Cytel: So taking a different direction, your paper on prediction for interim data monitoring of clinical trials is one of the foundations for East PREDICT. What was the motivation for this work?
SE: Interim data monitoring of trials is a very important issue. There are a number of advantages if you're able to conclude efficacy or futility of an intervention before the end of the trial. There are economic advantages and ethical advantages in that you may not need to expose patients to a treatment that might be inefficacious or toxic. Furthermore there are public health advantages in that you can get answers out to the medical community sooner.
"A formal evaluation of what the future might look like can aid decision making."
We had two concerns with the traditional interim monitoring practices. The first concern was that there's often an overemphasis on statistical significance and not enough attention to clinical relevance. We wanted an approach that would provide more focus on magnitude of effect so that you could assess clinical importance of the findings.
The second concern we had was that decisions are traditionally based only on the data in hand, but that could be augmented with a formal evaluation of what might happen if the trial were to continue into the future. You might make an assumption that the current trend that you're seeing will continue, or you might make more optimistic or a more pessimistic assumptions to try to understand what the trial result would look like in such cases. A formal evaluation of what the future might look like can aid decision making. We thus developed the predicted interval concept. Predicted intervals are essentially confidence intervals that could be observed were the trial to continue to its completion. We developed predicted interval plots (PIPS) as a tool to help DMCs make more informed recommendations.
Cytel: You have published a new textbook Fundamental Concepts for New Clinical Trialists. Could you tell us a little about the book and how it's been received?
SE: Sure. I've been teaching clinical trials courses here at Harvard for several years. Over the years I’ve played multiple roles in trials as a statistician on a protocol, or serving on data monitoring committees, or serving on FDA advisory committees. I began to collect and document my experiences and I used those experiences in my teaching to illustrate concepts in clinical trials. I decided to put together a book that discussed fundamental concepts in clinical trials and used personal experiences as illustrative examples. I now use the book in the courses that I teach.
It took many years to develop as it always had tertiary priority to more urgent projects. But this also allowed me to collect more experiences. I also wanted to be thorough and thoughtful in writing, and ensure that it was different from what other people had done. It is rather non-technical as other books can address specific statistical methodologies in great detail. I put together extensive discussions on certain topics- such as benefit-risk assessment, elements of trial design, and trial reporting which perhaps other books have not covered in detail. I teamed-up with a friend, Naitee Ting, who works in industry as we thought that would provide a complementary perspective on various issues.
I view clinical trials in four stages. There's the design of the clinical trial where there are fundamental concepts about randomization and blinding and control groups and what endpoints to use and what populations to study. The second stage lis what’s happening during the course of the trial such as data monitoring issues and DMCs. The third stage is statistical analyses and associated issues such as missing data, subgroup evaluation, and multiplicity issues. Finally, there is the reporting of trial results and publishing in the medical literature. This involves thinking about how to pull it all together and present a fair and balanced description and interpretation of the benefits and harms and the quality of life that was observed during the trial.
The book sequentially addresses these four stages: design, data monitoring, analysis, and reporting and publication. It's still a bit early to fully evaluate how it will be received. I've had some good comments from some students and colleagues and friends, but that's been a biased sample admittedly. I'm sure more data will come in soon. I believe, however, that it provides a helpful resource for people who are interested in clinical trials, and has unique sections that perhaps are not present in other books.
" The treatment of patients is not a single decision but a sequence of decisions over time whereby treatment adjustments may be made as new information such as patient progress, accumulates."
Cytel: Finally, what are future areas of research for you?
SE: I think there's still a lot of work to do with how to implement many of the benefit-risk methodologies and tailoring them to different disease areas. I have great interest in continuing that work. We are also working in the area of personalized medicine, evaluating how we can look at specific patient characteristics and then tailor treatment decisions to those individuals in order to optimize their outcome. We are also working on ways to evaluate and incorporate individual patient and clinician preferences. The way we feel about a particular toxicity or health state may vary from patient to patient. How can we bring in these preferences in a more structured and informed way? This will provide patients and clinicians flexibility in the decisions they make.
Along the lines of personalized medicine, we are evaluating dynamic treatment strategies, with emphasis on the word “strategy”. The treatment of patients is not a single decision but a sequence of decisions over time whereby treatment adjustments may be made as new information such as patient progress, accumulates. Thus, we’re studying dynamic treatment strategies that allow for adjustments to therapy along the way.
This happens in the antibiotics world in which a patient may come in and you treat them empirically based upon clinical presentation. This is typically the first major decision that's made. Two or three days later, one can observe the early response of the patient, whether they are getting better or getting worse. You may also get new laboratory information that gives you additional information about what the offending pathogens are and their drug susceptibilities. This provides a second major decision point, whereby treatment adjustments could be made based on the progress of the patient and other new information that was not available at the first decision point. Instead of evaluating each decision in a segmented way, we are linking these treatment decisions together into comprehensive strategies, and evaluating the strategies. Therapy adjustments are personalized in that each patient's adjustment is based upon their patient-specific data.
The last area I've been thinking about is clinical trials education. There may be novel ways to educate people about aspects of clinical trials.
Cytel: In terms of education for patients or in general?
SE: Both. We’re thinking about how can we help patients better understand the benefits and risks of an intervention so that they can make informed decisions. Patients may not have a scientific or research background to understand the typical manner in which research is presented with hazard ratios or this sort of thing. Are there ways in which we could simplify that for patients? Information could be presented to them in more thoughtful ways so that patients can comprehend and digest the information.
I'm also thinking about ways in which we educate the researchers of tomorrow, whether they are clinicians or statisticians or other people involved with clinical trials. With the evolution in technology, there may be ways to utilize the technologies to better educate them and to hold their interest about important concepts. There may be creative and modern educational delivery systems that we could use to educate the research community more effectively.
"I hope that in the end we're able to help patients get better treatments and enable clinicians to make better decisions."
Finally, what impact do you hope your research will have?
SE: Firstly, I hope that we'll be able to provide better care for patients, perhaps indirectly by finding better ways to design, monitor, analyze and report trials and diagnostic studies by being a bit more pragmatic. I believe that we can do this by finding ways to more closely link the patient journey and experience to our methodologies. I also hope that in the end we're able to help patients get better treatments and enable clinicians to make better decisions. Hopefully we can get more statistical people interested in these types of problems. This will help us make advancements in science.
Cytel: Thank you to Dr. Evans for these insights. We look forward to hearing more about your research in the future.
Learn more about Dr. Evans’ books:
Fundamental Concepts for New Clinical Trialists
Sample Size Determination in Clinical Trials with Multiple Endpoints
Group-Sequential Clinical Trials with Multiple Co-Objectives
Learn about Dr. Evans’ latest research below:
1) Evans, S. and Follmann, D. (2016). Using Outcomes to Analyze Patients Rather than Patients to Analyze Outcomes: A Step Toward Pragmatism in Benefit:Risk Evaluation. Statistics in Biopharmaceutical Research, 8(4), pp.386-393.
2) Evans, S., Pennello, G., Pantoja-Galicia, N., Jiang, H., Hujer, A., Hujer, K., Manca, C., Hill, C., Jacobs, M., Chen, L., Patel, R., Kreiswirth, B. and Bonomo, R. (2016). Benefit-risk Evaluation for Diagnostics: A Framework (BED-FRAME). Clinical Infectious Diseases, 63(6), pp.812-817
3) Evans, S., Rubin, D., Follmann, D., Pennello, G., Huskins, W., Powers, J., Schoenfeld, D., Chuang-Stein, C., Cosgrove, S., Fowler, V., Lautenbach, E. and Chambers, H. (2015). Desirability of Outcome Ranking (DOOR) and Response Adjusted for Duration of Antibiotic Risk (RADAR). Clinical Infectious Diseases, 61(5), pp.800-806
4) Evans SR, Hujer AM, Jiang H, Hujer KM, Hall T, Marzan C, Jacobs MR, Sampath R, Ecker DJ, Manca C, Chavda K, Zhang P, Fernandez H, Chen L, Mediavilla JR, Hill CB, Perez F, Caliendo A, Fowler Jr. VG, Chambers HF, Kreiswirth BN, and Bonomo RA. Rapid Molecular Diagnostics, Antibiotic Treatment Decisions, and Developing Approaches to Inform Empiric Therapy: PRIMERS I and II. Clinical Infectious Diseases, 2015.
5) Evans SR, Li L, Wei LJ. Data Monitoring in Clinical Trials Using Prediction. Drug Information Journal, 41:733-742, 2007.
6) Evans SR. When and How Can Endpoints Be Changed after Initiation of a Randomized Clinical Trial? Public Library of Science (PLoS) Clinical Trials 2(4): e18. doi:10.1371/journal.pctr.0020018, 2007.
7) Evans SR, Hujer AM, Jiang H, Hujer KM, Hall T, Marzan C, Jacobs MR, Sampath R, Ecker DJ, Manca C, Chavda K, Zhang P, Fernandez H, Chen L, Mediavilla JR, Hill CB, Perez F, Caliendo A, Fowler Jr. VG, Chambers HF, Kreiswirth BN, and Bonomo RA. Rapid Molecular Diagnostics, Antibiotic Treatment Decisions, and Developing Approaches to Inform Empiric Therapy: PRIMERS I and II. Clinical Infectious Diseases, 2015.
