Discussion 10
Happy holidays job analysts!
This is a very brief job analysis update (my present to you!).
1) A recent edition of The Industrial/Organizational Psychologist ("TIP") had a useful article on using Excel (which, as some of you have recently discovered, is an adventure!). It can be found at http://www.siop.org/tip/Oct03/20weiss.htm; the entire issue is available at http://www.siop.org/tip/Oct03/TOC.htm. You may also find Frank Landy's article an entertaining read: http://www.siop.org/tip/Oct03/19landy.htm. Talk about exciting job analysis!
2) As some of you know, on 11/3 a jury ruled that CHP did not discriminate in its hiring and promotions. According to a CHP representative at ESF, a big reason why they prevailed was because they had a job analysis that supported the use of their exams! (Note that this was for intentional discrimination only; an adverse impact decision by the judge had yet to be determined*)
3) Annette Roberts (HCD) raised an issue which I think deserves group feedback. Namely: What do you do when tasks or KSAs should be removed from consideration due to their ratings, but the SMEs are resistant and would like to keep them? (ignore for the moment the illogic that they want to keep something that they said isn't important, although certainly this would be a point of discussion).
Hope everyone is doing well! Have a wonderful holiday season.
Bryan Baldwin
Dept. of Justice
Hey Bryan, Happy Holidays to you too!
On the HCD question, did the SMEs rate the 'criticality or importance' of the task high, but the 'frequency' of performing that task low? If so, could that task be eliminated (when plugged into the formula on the Task Rating Summary Worksheet)?
Regena
Dept. of Mental Health
Brief answer to your question. Think about a police officer and his weapon.
The officer rarely draws his/her weapon, however, once the weapon is drawn,
it is extremely important that they know how to use it. Therefore, it would
depend upon the classification etc. Hope this helps
Bill
Board of Equalization
Hi there - well in answer to the question, it is very simple. The SMEs are
to assist us in determining the tasks, knowledge, and abilities most
necessary for testing persons who want to enter the classification. If
tasks fall our of the scope of rating due to their score - they are gone.
The SMEs do not tell the analyst what will be rated and what will not. The
items should be removed and the SMEs asked to rate the remaining tasks. If
they do not want to comply, they should be "fired". Otherwise, who is
running the exam? SMEs, SSRs and others do not run the exam and do not
have to respond to the appeals. Okay, I know you are all probably sick of
me saying this, but: It is YOUR exam, take control - it's YOUR delegation
you are risking.
Karen, Manager
Selection and Recruitment Unit
Department of Consumer Affairs
We discussed, in my office, the exact situation you brought up Bill. There may be a situation when the rating of that one task is critical, yet performed so infrequently, that it's eliminated when stats are plugged into the formula.
In the situation with the firearm, would you really want to eliminate that task, when the skill is so critical (even if the stats alone say to eliminate it)? I would discuss with the SMEs and review the ratings of those questionable tasks to determine just how critical the task(s) really are.
So....Bryan, what's the right answer?
Regena
Dept. of Mental Health
The key is "criticality" . If it is critical for performance, but
something that can be learned on the job and not necessarily critical to
"newbees" to the class - don't rate it. If it is critical for "newbees"
and oldees - rate it. However, it is the analyst that needs to make that
determintion based upon the job analysis. This would come out as part of
the analysis.
Karen
Department of Consumer Affairs
Here are my thoughts . . . and the way we are doing things here.
It is my understanding that the statistical formula is your starting point.
As Regina indicates, there are times when you perform a task infrequently,
but the job is their to do that task. In addition to the peace officer
scenario . . . think about accounting classifications. Tasks and KSAs
reflecting the year-end closing process typically rate low and can even be
eliminated if only a few individuals do this function. However, reconciling
the accounts at the end of the year is a very critical purpose (if not the
only purpose) of some jobs. In our process, we take the statistics back to
the SMEs and discuss those that may be eliminated. If there is a valid
reason to keep them in the mix, we document that in the analysis narrative.
The statistics are valuable, but they are not the absolute. To rely only
the outcome of your statistical data without additional input from the SME
could diminish the analysis outcome and subsequently the exam plan.
Barbara, Chief
Selection Unit
Department of Health Services
It sounds like we're all on the same page. This is a great example of
not letting mathematical formulas get in the way of common sense. The
idea behind the analysis is to get a true picture of what the key tasks
(and KSAs) are. We all know that the rating process is imperfect at
best; if a subsequent conversation reveals that SMEs believe strongly
that a task or KSA should not be removed, I think it's unproductive (and
unwise) to do so. The only caveat I would note is that this interaction
should be documented in the job analysis report (or whatever written
document is produced). When it comes time to develop the selection
method, there should be a frank discussion of what the ratings mean and
whether it would be appropriate to select based on this information or
if the candidates are likely to be trained on that KSA by the time they
need to draw a weapon, calculate year-end numbers, etc. I think the
WRIPAC method, as well as other job analysis methods, would not be
compromised by this type of analysis (in fact I think experts would
encourage this approach).
It might be instructive to change, say, one or two ratings, and see if
that changes the result--this may be additional justification for
retaining the task/KSA. Remember also that the numerical cut-offs are
guidelines, not absolutes, although it does make me wonder whether the
decision rule should be that the task/KSA has to meet BOTH cutoffs (so
that in a situation like this it would still survive because the other
value was high enough--i.e. criticality and frequency must be below a
certain number).
Great responses!
Bryan
And the plot thickens!! First, though, Happy Holidays to Everyone!! Hope the season finds you all well and in grand spirits!!
Now, for the thickening plot - there are really two issues at hand when we are interpreting the task and KSA ratings of our SMEs in the job analysis process. First, we want to establish an accurate picture of the job/job classification in terms of the essential (per ADA), critical tasks of the job. Second, we want to identify those KSAs necessary for successful performance of those essential, critical job tasks - and, as a little subplot, for exam purposes, we want to know of those KSAs important/required for job success, which are actually required day one. We make these two determinations separate and apart from each other to the extent that we use different ratings and interpret the rating results differently between the tasks and the KSAs.
In terms of analyzing the task ratings, we typically "define" critical as a combination of the importance of the task and the frequency with which it is performed. Thus, there are often tasks that are both important when performed and frequently performed. There are also typically tasks that are of lesser importance, but still frequently performed. And (as the pattern goes), there are also tasks that are extremely important when performed, but performed on a very infrequent basis (e.g., the police officer example of rarely drawing a weapon, but that event/task being critical when it is performed). So what does all this mean in terms of interpreting task ratings? It means that as we analyze the SME rating data for our tasks, we should only include in our task results those tasks that are deemed to be essential/important - whether frequently performed or not. This way we don't lose the "critical, but not performed often" tasks. Depending on the methodology used to conduct the job analysis, this does some analysis on our part. In the big picture, my guess is that SMEs are not usually concerned when tasks fall out as being either non-essential and/or non-critical.
I think the real issue with SMEs interpreting the job analysis results after-the-fact is when they question KSAs that have fallen out in the analysis process. So, what happens in the KSA analysis process?
Well, as we analyze the KSA ratings, we are (again for exam development purposes) wanting to identify those KSAs important for job success and required upon entry/appointment to the job/job classification. To this end, we should be gathering those two ratings for each KSA - an importance rating and an expected at entry rating.
Given the ratings, a KSA could be eliminated if it is deemed non-important to job success and/or not required upon entry. It is then this group of eliminated KSAs that can sometimes cause angst in our SMEs - the response back to the SMEs, though, should go along the lines of the KSA(s) in question not being important to successful job performance and/or not being required upon entry. Certainly a KSA could be critical to job success, but if we are going to train the incumbent on the particular KSA, then it is inappropriate to have it included in the exam process. Conversely, if a KSA is not important for successful job performance (whether possessed day one or not), then it doesn't make sense to include it in the exam process.
However, any KSA that is deemed important, but not required upon entry is still an important job qualification - it just doesn't have a place in the exam process as we normally conduct our civil service exams. The KSA could certainly be a basis for evaluating probationary job performance following training or job assignment experience, etc. So, for these KSAs, the message to the SMEs is really not that the KSAs are not important to job success, but that they are not required day one - they are somehow acquired post-appointment, but incumbents can still be held accountable for demonstrating possession of the KSA at some point other than in the exam process.
I think sometimes the issues with SMEs stem from the fact that as Exam Analysts, we are concerned with a very specific slice of the selection process, and they are looking at the long-term ramifications of what we are doing and saying. So when the message from the job analysis is that "these KSAs fell out and are not being considered further," we mean considered further only in terms of how far the exam goes. If the SMEs are hearing "further" as "forever," then that sometimes can cause problems. If we really tailor our message to say, "Based on the results of the job analysis, it was determined that these KSAs are not required day-one on the job; therefore, we won't be considering them in the exam process. However, they could still be appropriate in the hiring interview stage or during prob. or even during long-term performance evaluation." This little twist may actually go along way to allaying SME concerns.
As far as the "conversation" among us has gone, the point made about scale cut-offs being "guides" is well made, and there is much room for interpretation/analysis on our part in conjunction with SMEs - what we shouldn't do, however, is take data that very clearly indicates one thing and go a completely different direction simply on the basis of anecdotal, verbal SME input.
Wow, huge issue and let me take this opportunity to plug the January TestTalk session, which be covering these very issues, and a few broader ones, as well:
Session: Job Analysis Data - Now that We Have It, What Do We Do with It?
Date: January 21, 2004
Time: 8:30 a.m. - noon ** ½-Day Session **
Place: SPB, Room 150 - Auditorium
Take care, All!
Shelley
I seem to have gotten off the response thread at some point (either that
or Shelley has a secret plot to exclude me).
Just wanted to say that I think this is a great discussion that should
be continued--at Test Talk, at ESF, wherever we can fit it in. I think
it's important to make sure we're communicating clearly. I assumed in
this particular case that the verbal SME input was legitimate; I did not
mean to imply that other data should be thrown out, simply that it would
be wise to consider it as part of the larger analysis and not
automatically assume that numbers are sacrosanct.
The bottom line for me is it's a case-by-case situation. Sometimes
something hits you and you think, "Ya know, that makes sense. I better
think about this before moving forward", and other times you think,
"They're being resistant but I have confidence in the process." This is
just another one of those times when judgment is key.
Bryan
Print this page.Last modified: 12/4/2007
