"Sigurd Hermansen" <HERMANS1@WESTAT.COM> wrote:

>Before we try and convict the developers of the SAS Certification exams, it >seems appropriate to take a few moments and verify that the questions and >responses posted on SAS-L came from the official SAS certification exam >itself. I have seen questions from a employment screening exam and from a >Web course. Both included questions as bad or worse as the posted questions. >It does not seem possible to me that really bad questions could slop through >any systematic evaluation process.

I would not have thought it possible had I not taken one of the early set of questions offered on the web.

>Whatever the evaluation process, developers of the SAS Certification exams >should take a look at the questions from the 2003 SAS Bowl at the Seattle >SUGI. I don't know for sure who developed those questions, but I found them >relevant and challenging.

Even a few of those were arguably incorrect.

>Perhaps I found the questions challenging merely >because I could answer so few of them. The fact that the elite Teasipper >team from across the pond actually understood the questions clearly >separated them from the chaff (the SUGI Executive Committee, who reacted to >the questions as if Ray Pass were asking them in Navajo). Questions of the >quality of those for this latest SAS Bowl (forget the prior events) would >reliably identify truly advanced SAS programmers and developers. At least >ask those involved in developing the 2003 SUGI Bowl questions to review SAS >certification exam questions.

Yes, involving working SAS programmers outside of SAS Institute would be a good idea.

>I wonder who will require SAS certification, or at least give preference to >programmers or developers who have SAS certification. After interviewing >hundreds of candidates for SAS and other programming jobs, I'd say that an >experienced SAS programmer can classify a candidate >in about fifteen minutes with reasonable accuracy as a beginning, >intermediate, or advanced SAS programmer. Will the certification exam >identify those that might fool an interviewer? I still wonder whether >passing a certification exam reliably identifies competent programmers. Ian >Whitlock has suggested that a good certification exam would pose a problem >and require the examinee to write, test, and execute a SAS program to solve >the problem. The examinee would have SAS manuals, context help, and other >normal tools available for the exam.

That has potential, but the ability to write a short program under pressure may not correlate with the ability to write a complex program in a few weeks. Looking at previously written code is also a good idea, although of course you don't know how much of it was actually written by the candidate.

>I prefer Ian's method of testing. I >would not fly with a pilot that has passed a written certification exam but >has not demonstrated that he or she can actually fly an airplane.

I think I'd prefer both.

-- JackHamilton@FirstHealth.com Manager, Technical Development Metrics Department, First Health West Sacramento, California USA