LISTSERV at the University of Georgia
Menubar Imagemap
Home Browse Manage Request Manuals Register
Previous messageNext messagePrevious in topicNext in topicPrevious by same authorNext by same authorPrevious page (December 2006, week 4)Back to main SAS-L pageJoin or leave SAS-L (or change settings)ReplyPost a new messageSearchProportional fontNon-proportional font
Date:         Thu, 28 Dec 2006 18:04:54 -0800
Reply-To:     Cyclotron <dramitgupta@GMAIL.COM>
Sender:       "SAS(r) Discussion" <SAS-L@LISTSERV.UGA.EDU>
From:         Cyclotron <dramitgupta@GMAIL.COM>
Subject:      Comparing two ROC curves after bootstrap
Comments: To:
Content-Type: text/plain; charset="iso-8859-1"

I am trying to compare the bootstrap corrected (200 replicates) c-statistics of two logistic regression models (one without and one with a marker)

****************************************************************************** The code I am using is :

/*CREATING THE BOOTSTRAP REPLICATES*/ proc surveyselect data=sasuser.hgfexp out=boot seed=150 method=urs samprate=1 rep=200; ID outcome age sex race marker; run;

ods listing close; /*BASE MODEL: Logistic regression on the bootstrap replicates*/ proc logistic data=boot; model outcome = age sex race; ods output association=out1; by replicate; run; ods listing;

/*Keeping only the c-statistic*/ data out1(keep=c); set out1; if label2="c"; c=nvalue2; run;

/*MODEL with marker: Logistic regression on the bootstrap replicates*/ ods listing close; proc logistic data=boot; model outcome = marker age sex race; ods output association=out2; by replicate; run; ods listing;

data out2(keep=c); set out2; if label2="c"; c=nvalue2; run;

/*Changing the name of c (from the second model) to cmarker*/ data out2; set out2; cmarker = c; drop c; run;

data comp; merge out2 out1; run;

/*Calculating the difference between the c-statistics*/ data comp; set comp; diff = cmarker - c; run;

/*T test for the hypothesis that the difference between the c-statistics is zero*/ proc ttest data = comp; var diff; run; ******************************************************************************

When I run the above model the difference between the c-statistics is like 0.1 but it is statistically significant using the t-test.

This does not feel right. Also the T-test is being done on a database with 200 values for the difference between the two statistics and thus will always be significant because of the sample size. When I run the same analysis with less number of replicates (50) then the t test is not significant as the standard error is higher.

What is the correct way to compare the two c-statistics here?

Any help will be greatly appreciated. Thanks Amit

PS: I looked at SUGI27: Paper 248-27, Use of the ROC curve and the bootstrap in comparing weighted logistic regression models. In that paper in the end this is how they calculated the difference between the curves i.e. by running a t-test. Am I mis-interpreting the paper? Should the df for the t-test be different?

Back to: Top of message | Previous page | Main SAS-L page