Cytel recently hosted a very well-attended and engaging webinar on the topic of “Estimands, not just a statistical issue” presented by Paul Terrill, Associate Principal of Strategic Consulting at Cytel.
The webinar covered a range of issues from what is an estimand to how to structure early discussions on estimands.
In this blog, we are happy to share the replay of the webinar as well a summary of Q&As that arose on this very important topic. For an introduction to the topic, check out our previous blog post 'Estimands 101 with Mouna Akacha'.
Would you consider estimands in all the phases of clinical development?
This is a fairly common question because although the ICH E9 guideline is more focused on confirmatory studies, it is relevant to all phases of studies. The estimands guideline is no exception to this, because no matter the phase of development we are still trying to measure something, estimate something, and have an objective. The ideas and concepts behind estimands are relevant whenever you're trying to estimate something; you have to understand what it is you're trying to estimate and what the treatment effect of interest is.
Can you have more than one estimand per study?
Yes, you certainly can. In fact, we are going to have to moving forward because we will need to specifically state what it is we're trying to estimate and what stakeholder the estimand is for. We will generally have what we term a primary estimand, just like we are used to having a primary objective, primary endpoint, and primary analysis. However, it could be that we also want one estimand for a regulatory authority, versus a different estimand for a Health Technology Assessment (HTA) and another for our own internal understanding. What’s particularly nice about the framework is we can specify how these estimands are different, while enabling us to have more than one. It also means that we are likely to generate more outputs since answering a different estimand brings with it a different set of analyses. But in summary, yes the study can have more than one estimand within the protocol.
Should we define estimands for secondary objectives?
If you don't define estimands for your secondary objectives, you end up with exactly the same problem that you will have if you don’t define them for the primary objectives - that it's not clear what it is you're measuring and what you're trying to understand. Strictly speaking, you need to understand every objective and every question. To understand every objective and every question, you need to understand what it is you're trying to estimate. Then, by its very nature, you need to define the treatment effect and how to take into account the inter-current events. If you don't do that for your secondary objectives, you run the risk of not clearly defining what that objective is or coming up with a correct answer to that objective.
How should we practically define estimands to meet the needs of multiple stakeholders, and where in the protocol should this be placed?
I suggest to include some of the estimand framework within the overview of the objectives in your protocol. This means you may need to include multiple objectives in your synopsis if you're gearing your study to answer different questions. So, I would include it in the wording of the objectives and then you can either choose to put more detail within the synopsis or add it later in the protocol. I expect we will move away from protocols which have an ill-defined objective, a list of endpoints, a list of summary measures and then maybe some handling of missing values in separate places within the protocol. These different sections can be very hard to tie together, whereas by pulling a single paragraph together, that covers all of these aspects (or multiple paragraphs if you are addressing different questions) is a helpful exercise.
What are the pros and cons of the estimands framework compared to the PICO framework from the HTA perspective, specifically with aggregated data in a network meta-analysis (NMA) context?
(PICO = P – patient, problem or population. I – intervention. C – comparison, control or comparator. O – outcome.)
PICO is a well-established framework and so many people are familiar with it. It is a pity that PICO isn’t mentioned in the E9 addendum. In my view, the PICO and estimands framework are generally trying to do the same thing by discussing the different items you need to consider and take account of. These concepts (i.e. the concepts behind estimands) are just as relevant for NMAs and for when working with aggregated data. The main point is that even if using aggregated (study level) data in a NMA, in principle at least we would want to make use of results that were measuring the same thing (same estimand / same attributes = population, variable, summary measure and accounting for inter-current events) so that we are combining apples with apples. Or at least combine as similar apples as we can. Of course this may not be achievable if we only have the study level results and not individual patient level data and/or if the aggregated data does not specifically describe what treatment effect was estimated. This isn’t a new problem that has been highlighted by ‘estimands’, rather it’s just another way of seeing the same problem that NMAs have.
Missed the webinar? Click the button below to access the replay.
Related resources:
Hernán MA, Robins JM (2019). Causal Inference. Boca Raton: Chapman & Hall/CRC, forthcoming. https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/
ICH Training material available for download here:http://www.ich.org/products/guidelines/efficacy/article/efficacy-guidelines.html#9-2
