I am interested in folks opinion of using the following procedures to report on the effectiveness of an intervention. I work for a non-profit and have limited time. I have found the following acceptable when reporting to stakeholders and successful when submitting grants, however, I've recently been told by a professional evaluator that my process is terribly flawed.

I utilize the following: 1) paired sample t-test applied to 3 subscales of pre & post measures obtained from an instrument with established reliability and validity. I report an n for each subscale. When I run the analysis, I eliminate records if any of the questions within the subscale were not answered. 2) frequency of Dept. Juvenile Service level of involvement pre intervention and 12 month post intervention. 3) Satisfaction survey at completion of intervention.

The evaluator has informed me that: 1) I cannot use the paired sample t applied to calculated scores for each subset. I must report an n and the frequency of each question within each subset. I would then be able to identify the particular areas that have had the greatest impact. This approach seems very cumbersome although it would be useful if I were using the results of this examination for program improvement. 2) It was suggested that I modify the questions within a subset when I don't think the question completely captures the information I need. I am uncomfortable with this suggestion because wouldn't the reliability and validity of the question be lost? and how then would you report the results with confidence?

I would very much appreciate all comments.

Madeleine Ryan, MPA Grants Specialist 410-876-2500 ext. 27 maryan@ccysb.org

Confidentiality Notice

The information contained in this electronic message is legally privileged and confidential under applicable law, and is intended only for use of the individual or entity named above. If the recipient of this message is not the above-named intended recipient, you are hereby notified that any dissemination, copy or re-disclosure of this communication is strictly prohibited. If you have received this communication in error, please notify the sender at the Carroll County Youth Service Bureau at the e-mail address above or at (410) 876-2500, and purge this communication immediately without making any copy or distribution.