Monte Carlo Simulations II: Reassessing Strategic Options During an Interim Look

Midway through a trial is a terrible time to realize that you need a new strategy to complete the study. Sadly, it is typically midway through a trial when drug supply, patient recruitment and budget all tend to deviate from the planned development path. Sometimes this is because the initial plan utilized idealized assumptions, (i.e. non-random patient enrollment), which failed to give the desired ballpark estimate of timelines and resource constraints. Other times, responding to unexpected operational or statistical challenges might have proven difficult due to inflexible trial designs [1].

We have spoken in some depth about how thorough planning and room for flexible decision-making can avoid some of these potential difficulties [1] [2] [3].  However, sometimes the specter of trial discontinuity arises anyway. Here is a scenario which recently confronted the Cytel Consulting Team.

Challenge: Can we complete this trial? 

Our client’s trial was so far behind on recruitment, they were beginning to consider whether trial completion was a feasible option given their operational constraints. The problem was not a shortage of strategic options. Rather, the problem was the difficulty in assessing which of these strategies, and in particular which combination of strategies, would lead to trial completion in an efficient manner. If they could not find a reasonable time-frame in which to complete this trial, discontinuing the trial would have proven a better use of their resources. 

According to several studies, many sponsors likely find themselves in comparable situations at one time or another. A recent JAMA paper, for example, cited enrollment as the leading cause of Phase 3 trial discontinuity, with over 30% of discontinued trials citing it as the reason for incompletion. [4] The Cytel Whitepaper on Adaptive Clinical Trials also points out that 10% to 30% of clinical trial sites reportedly fail to recruit a single patient [5].

Cytel’s Simulation Based Solution

• Step 1: Our team built a model that captures how various factors like site activation and improved screening affect patient enrollment rates. Using the model and historical data, we determined the degree to which these affected enrollment rates independently and in combination.
• Step 2: Using this model, our team developed software that allows sponsors to evaluate the effects of different possible interventions (i.e. lets sponsors visualize how changing one or two inputs will affect patient enrollment rates across the study.)
• Step 3: Using this software and Monte Carlo Simulation techniques, our team created simulations that showed the likelihood that certain strategies would succeed. Monte Carlo simulation techniques begin with a set of assumptions about the trial in question, (e.g. screening and enrollment rates, country-specific variations,) and then project what will happen in thousands of instances, if these assumptions are true. By determining what would occur 95% of the time, 50% of the time, and so forth, Monte Carlo simulations give a sponsor a sense of the likelihood that different strategies will succeed.  
• Step 4: Using the data from Step 3, our client was able to determine which strategies are most likely to succeed, and come up with a new strategy with which to move forward.

Lessons Learned

Our client’s situation may seem more familiar to some than to others. In general, though, their willingness to use simulation techniques to reassess their strategic path midway through a trial is one that all adaptive clinical trial sponsors ought to consider.

The fact is that much information is available midway through a trial that was not available before. Why not use this information to reassess strategy? 

For example, suppose that at the beginning of a trial you expected there to be a 10% chance that a certain trial site would begin to screen patients at a given rate. However, towards the end of the trial you realize that the site has in fact achieved this screening rate. In other words, this 10% chance has become a 100% reality – the trial site has in fact begun to screen patients at a faster (or slower) rate than you had reason to believe initially.

Why not come up with a superior strategy with the knowledge that you have right now?

Perhaps you can complete your trial as you had always planned, but with greater data and more insights about your trial’s development path, superior strategies can present themselves that would not have been available earlier in the trial.

   How to construct Monte Carlo Simulations:

In some cases you might want to call in Statistical Consultants during an interim look, to help you avoid the worst-case scenario. However, Monte Carlo Simulations are also available through Cytel’s Enforesys®, the first software of its kind to perform such techniques specifically for clinical trials.