3 Strategies to Combat Inferential Conservatism in Small Sample Sizes

June 25, 2015

Chris-Corcoran-photo-4web-031814When conducting a clinical trial with small or sparse data sets, statistical methods meant for large sample sizes may fail to obtain an accurate interpretation of data. This is where computationally challenging exact methods often come into play.

Exact methods, however, are inferentially conservative in the sense that due to small sample sizes, the actual Type 1 error rate is often smaller than the nominal (intended) rate [1]. There exists an array of strategies to combat this troublesome feature of exact tests, each of which varies along the parameter of computational complexity.

Request StatXact 11 In a June 2015 webinar, Professor Chris Corcoran of the University of Utah, illustrated three strategies to reduce conservatism along with simple software techniques to compute these results.  He considers these strategies with a view to optimizing type 1 error control, maintaining high statistical power, and ensuring rapid computation.

Chris considers the following strategies to reduce conservatism.

Strategies to reduce conservatism:

  • Blaker’s Method (Blaker 2000): Refines exact distribution to enable actual Type 1 error rate to come closer to nominal, thus improving statistical power.
  • Bayesian Approach for Two Proportions: The proportions are treated as prior distributions, while the difference between them is the posterior distribution.  
  • Bias Correction of MLEs (Firth 1993): Corrects bias in maximum likelihood estimators, so that one can employ a more robust estimator without the computational challenges of exact methods.

The video below shows Chris employing these strategies to clear, believable examples, showcased using Cytel’s StatXact 11 suite. Click to watch. 

Chris is a Professor of Statistics at Utah State University, with a doctorate in Biostatistics from Harvard University. He has worked closely with colleagues at Cytel Software for nearly 20 years in developing new tools for StatXact and LogXact, including exact tests for correlated data, bias-corrected models for logistic and Poisson regression, and permutation-based tests for modern genome-wide association studies.
Related Items of Interest
[1] Agresti, Alan, and Anna Gottard. "Reducing conservatism of exact small-sample methods of inference for discrete data." Compstat 2006-Proceedings in Computational Statistics. Physica-Verlag HD, 2006. 245-260.
[2] Blaker, Helge. "Confidence curves and improved exact confidence intervals for discrete distributions." Canadian Journal of Statistics 28.4 (2000): 783-798.

[3] Firth, David. "Bias reduction of maximum likelihood estimates." Biometrika 80.1 (1993): 27-38.