"Cacialli, Doug" <Doug_Cacialli@URMC.Rochester.edu> replied: > Thank you for your concern. I recognize the dangers in reporting solely > significant results as well as any of us, and I have voiced them to the > powers-that-be. The powers-that-be are seemingly unconcerned. > > I was considering outright refusal to print only the significant results. > But my principles are less important to me than getting paid. I have some > nasty habits I need to feed (habits like, eating, and living indoors : -)

Perhaps it's time to take a lesson from Dogbert. Tell the Powers-that-be that you talked with some statisticians, who said their idea will not work. Say that the stat-geeks insisted that without a full accounting of what was significant and what was not, you won't be able to really tell what is 'significant' because of problems in computing the real experiment-wise error rates. In other words, if you have three significant cases out of three tests, you have a different likelihood of such an occurrence than getting three significant cases out of three thousand tests. The 'true' experiment-wise alpha is not that nice 0.05 if you perform three thousand tests with alpha=0.05 in each. Show them the Bonferroni calculations (which are admittedly over-conservative and assume complete independence) so an upper bound to the probability of making at least one Type I error becomes

1 - (1-0.05)^n

which equals (for n=3000) a frightening 1.0000000000000000000000000 likelihood of at least one Type I error. (The probability of *no* Type I error under the Bonferroni assumption is about 1.5E-67 .)

Then ask them whether they want to control the CER, the EERC, the EERP, or the MEER in order t make their experiment more reasonable statistically. (The explanations for all of these are in the SAS docs, under PROC GLM.)

HTH, David -- David Cassell, CSC Cassell.David@epa.gov Senior computing specialist mathematical statistician