A few weeks ago I wrote about new research conducted by Cytel statisticians, on the challenge of conducting meta-analyses for rare events. Studies involving rare events are often underpowered. Meta-analyses aggregate insights across studies, thereby enabling rare event studies to yield new insights to public health experts.
Here I discuss a new paper by Cytel scientists and collaborators, on how to manage such heterogeneity using CLRPerm, a permutation based approach utilizing conditional logistic regression. Simulations show that CLRPerm is better able to manage Type I Error than several other competing methods.
Challenges to Rare Event Meta-analyses
Meta-analyses generally rely on assumptions that will not hold for small sample studies or those with sparse data. Therefore, when conducting meta-analyses, these studies are typically excluded. Unfortunately, this tends to bias the meta-analyses, particularly for therapeutic areas where rare events are quite common.
Studies which conclude with zero observed events still reflect insights about treatment effects. This is even more so the case for studies with zero observations in one arm but not others. Yet both these kinds of studies are typically excluded from meta-analyses, thereby skewing the data and making the meta-analysis biased. Methods that adjust for this by adding an incremental non-zero ‘correction’ to the observed events are also noted to result in biased results.
A second difficulty, and the primary subject of the new research by Cytel, confronts the difficulty of dealing with heterogeneity in the studies used for meta-analyses. Heterogeneity often refers to differences in study parameters, populations and protocols, that require careful and technical expertise during processes of aggregation. Here, the heterogeneity refers squarely to heterogeneous treatment effect. Put more bluntly, the meta-analyses is actually aggregating different treatment effects about the same disease to shed light on safety and adverse events. Those conducting the meta-analyses can therefore use ‘Fixed Effects’ methods which assume that the meta-analysis is measuring a single treatment effect, or they can use ‘Random Effects’ methods which capture the heterogeneity.
CLRPerm: The New Permutation-based Method
Given the above, it is important to note certain features of the permutation based method proposed by Cytel Senior Vice President of Product Development, Dr. Pralay Senchaudhuri and his collaborators.
He and his collaborators have discovered a method which:
Does not rely on large sample approximations;
Does not use continuity corrections that often bias meta-analyses;
Does not use either a Fixed Effects or Random Effects method.
While the permutation-based method might seem most analagous to a random effects method, the method makes use of a proxy for heterogeneity variance, making it ‘not strictly random effects.’ The method in question begins with a conditional logistic regression framework, which is then extended to account for heterogeneity.
CLRPerm is computationally intensive but is far superior to competing methods at managing Type I Error. Click on the link below to learn more:
About the Author of Blog:
Dr. Esha Senchaudhuri is a research and communications specialist, committed to helping scholars and scientists translate their research findings to public and private sector executives. At Cytel Esha leads content strategy and content production across the company's five business units. She received a doctorate from the London School of Economics in philosophy, and is a former early-career policy fellow of the American Academy of Arts and Sciences. She has taught medical ethics at the Harvard School of Public Health (TH Chan School), and sits on the Steering Committee of the Society for Women in Philosophy's Eastern Division, which is responsible for awarding the Distinguished Woman in Philosophy Award.