Cohen's Kappa to find Measure of Agreement: How accurately do pregnant women report whether they smoke?

This analysis is taken from the research of Mark A. Klebanoff, Richard J. Levine, Cynthia D. Morris, John C. Hauth, Baha M. Sibai, L. Ben Curet, Patrick Catalano, and Diana G. Wilkins in their article "Accuracy of self-reported cigarette smoking among pregnant women in the 1990s", Pediatric and Perinatal Epidemiology volume 15, pages 140-143.

In this study, 104 women were asked whether or not they smoked, and also gave either a serum or a urine sample, or both. In order to determine how many of these women truthfully stated their status as non-smokers, the concentration of cotinine in the women's serum and/or urine samples was examined. Of these women, 35 provided both serum and urine samples. These samples were used to determine whether both types of samples classified women the same way reliably. A woman was classified as a smoker if her urine had a concentration of cotinine above 500 ng/mL or if her serum had a concentration of cotinine above 10 ng/mL. The results are listed below:

In order to measure how closely the two classifiers agree, we will compute the value of Cohen's Kappa. The value of Kappa estimated by StatXact's exact procedure is 0.80 (P=0.0001, 95% CI : 0.54 to 1.00). These results indicate strong agreement between the two classifiers.

Having established that the two classifiers agree, we can combine the data of all 104 women to form the following table.

Now to see how well women's self-reported smoking status matched the cotinine analysis, Cohen's Kappa is calculated for the above data. The estimate of Kappa for the above table from the exact procedure in StatXact is 0.72 (P<0.0001), with an exact 95% CI of (0.52, 0.92), indicating a strong agreement. Only 5 out of the 91 (5.5%) who reported non-smoking actively smoked. This percentage is very close to results from a similar study 30 years ago, suggesting that pregnant women in the 1990's did not feel additional pressure to lie.