Selection Bias for Treatments with Positive Phase 2 Results with Simon Kirby

October 18, 2018


In this blog, we talk with Simon Kirby, former Senior Director at Pfizer, about his upcoming presentation at Cytel’s East User Group Meeting on 14th and 15th November at Merck Darmstadt, in Germany. Simon will address the topic of Selection Bias for Treatments with Positive Phase 2 Results and in this blog he explains why this is a key topic of particular relevance for pharmaceutical companies in today’s climate of accelerated development. He also talks with us about his career in statistics, current research, and his book Quantitative Decisions in Drug Development.

Cytel: Could you give us a little bit of background about your interest in statistics and how you came to work in the field?

Simon Kirby (SK): I became interested in statistics at school, as part of my Maths A-level, although I studied economics for my undergraduate degree at university and returned to statistics a bit later with an MSc. Doing the MSc confirmed that I enjoyed the subject, and I went on to make it my career. I was attracted to the idea of working with something applied, that involves real data and where the potential usefulness is reasonably direct, as opposed to pure mathematics which could seem somewhat abstract.

Cytel: How did you first become involved in the pharmaceutical industry?

SK: When I finished my MSc I was initially employed at the Applied Statistics Research Unit at the University of Kent, Canterbury as a research assistant. That was my first involvement with the pharmaceutical industry and clinical trial projects. The Applied Statistics Research Unit had been fairly recently set up and I worked on projects for various clients during my time there. I then moved on to a job for one of those clients, Revlon Healthcare UK limited.

Cytel:How did you find the transition from studying statistics to working in an applied environment? 

SK: A typical MSc course has to cover a huge amount of material, often in a single year, and you spend a lot of your time working through this material with the aim of passing the final exam. I recall at the time that a lecturer said to me " you never really learn statistics till you go and use it", and I found that this proved to be very true. It’s one thing looking at formulae and studying in principle how you might do something, but totally another to apply those statistical techniques to real data and navigate all the practical problems that usually entails. There is quite a leap from an academic course to actually becoming an applied statistician, and it takes some years to fully appreciate all of the aspects required. 

Cytel: Is there any specific advice you would give to statistician starting out in the pharmaceutical industry?

SK: In my opinion, it’s about getting as much experience and talking to as many people as you can. Pfizer used to have a rotation system that took MSc graduates through a two-year program going around different parts of the company from research through to early and later phase development. It was a training ground whereby the participants could see the various dimensions of work across the company. It’s important to try and get that breadth of experience reasonably early on in your career.

Cytel: Can you tell us about the topic you are discussing at the East User Group Meeting in November “Selection Bias for Treatments with Positive Phase 2 Results?

SK: This topic has been around a few years and it first came to my attention with a paper by Sue-Jane Wang, Robert O’Neill, and HM Hung. (1) Before that, I hadn't fully appreciated that when you select good treatments from phase 2 you’re building in a bias simply because of the fact that you've selected the positive results. We wrote a fairly similar paper a few years later. (2) Although that work was done, I'm not sure how well appreciated the problem is. In a way, the issue is potentially getting more acute again, as companies bring greater pressure to bear on their staff to accelerate drugs through development.

For instance, when I first started at Pfizer you might have quite a number of phase 2 trials and so you'd be reasonably sure what a treatment could do. Nowadays you can find that companies are rushing ahead, and it’s quite possible they do only one phase 2 trial and then want to accelerate into phase 3, using a tight selection criterion to determine whether or not they are going to advance that treatment. If you do that sort of thing then you risk the problem of introducing bias to your phase 2 result and getting an overly optimistic view of what you will achieve in phase 3. In summary, I don't think the problem is sufficiently appreciated and I think the way that companies are now working perhaps makes the problem bigger.

Larger companies with more resources may try to look outside at competitors’ results that might be similar to gain corroborative evidence, or they might do modeling to try to incorporate other data and soften the effect of this bias. If you don't have other data, or if you are a smaller company with fewer resources then the potential for bias is perhaps greater compared with those who have access to data that they can add to what they already see.

Cytel: In practical terms, what do you hope that the people that hear your talk will come away with?

SK: I’d like them to come away with a greater sense of caution about simply extrapolating the phase 2 results to assume that that's exactly what they'll see in phase 3. Also, to gain a greater awareness that what they observe may be positively biased, or may be overly optimistic and make some allowance for that. 

Cytel: What are you working on now in terms of new research or publications?

SK: We are submitting a paper on the topic of the talk, selection bias, to the Pharmaceutical Statistics journal and we are hoping that it will be published. I tend to work closely with a former Pfizer colleague Christy Chuang-Stein who now works as an Independent Consultant and I'm also currently working with David Li so it’s quite possible we will carry on the collaboration after the paper hopefully gets accepted for publication.

Cytel: Could you give our readers an overview of your book Quantitative Decisions In Drug Development?

SK: The book draws an analogy between drug development and a clinical diagnosis problem. When you go to a doctor he has to work out what problem you have. When you've got a drug or potential new treatment the problem is working out is this product good enough? Is it good enough to be placed on the market and be commercially successful? It's a similar problem to diagnosing a disease as you are accumulating evidence continuously throughout drug development, and you get a better and better view of what the product can do.

The book takes the reader through the various stages of drug development, principally through phases 2 and 3 and looks at different available ways that you can assess how well the treatment is doing so far, gradually building this evidence as you move through drug development.

Cytel: What feedback have you had on the book?

SK: Generally, feedback has been very positive. It very much summarizes what we were doing at Pfizer, so it does represent a big company’s view of drug development. In that respect, it's useful for other big companies to look at, to see if they are doing similar things, or on the other hand, if you work for a smaller company you may not have been exposed to that sort of thinking. It's an attempt to put down on paper an overview of drug development as was current at Pfizer in 2017 and that probably isn't much changed just a year later.

Cytel: Where is the book available if people wanted to obtain a copy?

SK: They can get it from the publishers Springer Verlag, Amazon or from most booksellers.

Cytel: We look forward to hearing more insights from Simon at the EUGM in November.



1) Wang, S. J., Hung, H. M. J., O'Neill, R. T. (2006). Adapting the sample size planning of a phase III trial based on phase II data. Pharmaceutical Statistics 5(2), 85–97.

2) Kirby, S., Burke, J., Chuang-Stein, C., Sin, C. (2012). Discounting phase 2 results when planning phase 3 clinical trials. Pharmaceutical Statistics 11(5), 373–385.


Simon Kirby

About Simon Kirby 

Simon Kirby has recently retired after working for Pfizer for almost 20 years as a Principal Statistician then as a Clinical and Therapeutic Area Statistics Head before latterly being a member of Pfizer’s Statistical Consulting Group. Before working for Pfizer, Simon worked for Revlon Healthcare, Rothamsted, The Institute of Food Research (UK) and Liverpool John Moores University. Simon holds undergraduate degrees in Economics and Mathematics and an MSc and PhD in Statistics.



About the East User Group Meeting: Innovations in Clinical Trials Symposium & Hands-on East Training

Since 2011, the Innovations in Clinical Trials Symposium & Hands-on East Training has brought together industry experts, thought leaders and applied statisticians to discuss the future of clinical trials. This annual meeting guides the development of the industry's leading clinical trial design software. Join your peers at Merck in Darmstadt, Germany to see the newest developments in East.

Register today 

Register for both days and save over 30% - click the button below to review the full agenda and to register.