Q: What should be done when subject matter experts indicate that the measurement
of a KSA should be pass/fail, but we believe doing so is not making the best use of
the test - that it ignores important score differences?

A: Well, this is an interesting query and one that begs a broader question
in terms of how we interact with our SMEs as selection professionals
when we have occasion to somehow question their input at any point in
our dealings with them (e.g., during job analysis, during examination
development, during examination administration...). We can save the
broader application for a subsequent conversation, though - if we focus
on Bryan's question, here's my thought:

If an SME group indicates that a specific KSA should be pass/fail and
you believe that the KSA should be weighted (or at the very least
question the strict pass/fail interpretation), then there are probably a
couple of actions that should be pursued:

1. Determine if the SMEs understand that a KSA can be both pass/fail
and carry a weight - for example, basis mathematical skills... We could
assess candidates' mathematical skills in a weighted fashion (e.g., the
KSA carries a weight of 20%) and also set a pass point on a selection
procedure so that candidates have to demonstrate math skills at a
minimal level (in order to pass, hence a pass/fail requirement), and
then those candidates with greater/better developed skills will receive
a higher score, as their increased math skills will allow for them to
score that much higher than another candidate who demonstrates only
minimal math skills, and that higher score will be factored into their
final score/rank in the process. So that's really question one - is it
that more of the KSA won't contribute to greater job success, or is it
that the SMEs are unclear how the notion of pass/fail and
weighting/ranking works...

2. Then, we would next want to discuss our position/our thoughts with
the SMEs to the extent that we can explain to them why we think the KSA
should carry weight. Taking our math skills example from above, we
would need to pose a series of questions to our SMEs: will more
accuracy in completing "target" math calculations result in better job
performance? will one's skill to do more complex calculations yield
better job performance? etc... It's really a matter of discussing the
specific implications of their decision with them relative to the impact
that their decision will have on our scoring/measurement of candidate
qualifications. This discussion is probably very similar to how we
explained the notion of ranking at the outset of this "exercise" (i.e.,
this data collection point in our job analysis process), but it's always
a good idea to make sure that the SMEs are aware of how their
ratings/how their actions will affect the final outcome of our
examination development and administration activities.

3. If we've covered the discussion points outlined in #1 and #2 above
and the SMEs are still stuck on pass/fail, no weight for the KSA and we
still have concerns over this decision, then it's a matter of
determining an appropriate course of action depending on the
circumstances of the event:
a. Are all SMEs advocating pass/fail, or do we have a split decision
that just weighs slightly heavier toward pass/fail? Perhaps we can
"sway" in an appropriate/job-related fashion one or some of the SMEs,
swinging the results more in favor of weighting.

b. Is this an instance where we really do doubt the input of the SME
group? Perhaps we want to run the particulars of the situation by one
or two new/add'l SMEs to get their take. The smaller this initial group
of SMEs, the more that we could be concerned with coercion behind the
scenes (e.g., is one supv. really riding rough shot over the incumbent
participants? are the incumbents too timid to speak up?). Some of
these issues, if present, could actually be indicative of greater
problems with the process, but we'll keep it simple for this discussion
and assume that any oddity is present only for this one KSA in
question.

c. Depending on the job analysis methodology that we're using, we
might have some latitude as the selection professional to make a call in
favor of weighting the KSA. However, we should have some basis of SME
input for doing so. If we get into the habit of overriding SME
determinations at any point in the process (including job analysis, exam
development, exam administration), then we are really sacrificing the
ultimate value of our SME input as the basis for the job-relatedness of
the process.

I do think that with a thorough discussion with the SMEs, an issue such
as this one can be resolved - either the SMEs will see and agree with
our point, or we will come around to the rationale for their position.
The "iffy" piece of this puzzle is when we have SME input that says one
thing, and we go and do something different - that arbitrary change in
direction really compromises the integrity of an SME-based process.

Hope this helps - sorry for length, but big issues deserve big
answers!!

Have a great week, everyone!!
Shelley