Put your scale where your money is (or isn’t)
I had the opportunity to listen in on a web broadcast recently where Marc Effron was the featured speaker (as part of his “One Page Talent Management” approach, which I have not read as of yet), and the topic was advertised as, “Why 360’s Don’t Work (and What You Can Do About It.)” I found it very interesting and thought provoking, as did others on the call judging from the questions that were submitted during the call. In fact, I listened to it twice! Perhaps his most controversial position is his support for using 360 for decision making, which I support under the right conditions. I plan to write a couple blogs regarding a few of his positions regarding 360, and I hope Marc will comment if he feels I have misrepresented him and/or am incorrect in some way (which is definitely a possibility).
Marc proposed a number of approaches to 360 that he believes make the process more effective and efficient. One that caught my attention was his approach to the rating scale. In fact, it motivated me to submit a question during the session, and I will discuss his response as well.
Marc spoke of a “Do More/Less” scale that he uses that ranges from “Do Much More” at one end, to “Do Much Less” at the other end, and “Don’t Change” as the midpoint. I have seen a presentation where it is a 5 point scale, but I could easily see a 7 point scale as well.
During the web cast, as he was describing this scale, I believe he said, “We don’t care how good or bad you are.” In other words, he is proposing an “ipsative” approach to measurement (if I remember my graduate training correctly) where the focus is within-person ability (versus “normative” which is between-person comparison). In this context, the ipsative scale acknowledges that we all have a stack ranking of abilities from best to worst, regardless of how well we perform in comparison to others. In a development focused process, this has great appeal in communicating that we all are better at some things than others, and we all have a “worst at” list even if we think we are still pretty good at those things relative to others.
It seems arguable as to whether all raters use the “More/Less” scale as an ipsative scale. I am assuming that Marc intends it to be used that way based on his “don’t care how good or bad you are” comment. It would be nice if the instructions to the rater reinforced that point (i.e., don’t compare this person to others), and maybe they do. There are other ways to generate within-person rankings, such as paired comparisons and check lists, which seem more direct but probably have their own drawbacks (I have never used them, so I am no expert).
I see ipsative approaches to 360 rating scales as potentially being fantastic solutions for “development only” processes where users are forbidden from using the data for decision making (or so they say). Many of us know of supposed “development only” programs where the data is being used for decision making in various ways, creating potential inconsistency, unfairness, and misuse. If these companies used an ipsative scale such as Marc’s, that would theoretically prevent them from using it for decision making since the data is totally within-person and inappropriate (or worse) to use for comparing performance results across employees.
The problem with Marc’s situation is that he IS using this scale for decision making. So that was my question to him, namely how can you use a nonevaluative (ipsative) scale to make comparisons (i.e., decisions) among employees? His response was basically that, a) the “Do More” list is generally indicative of areas in need of development, and b) the 360 results are supplemented by other data . Point A seems to fly in the face of the “don’t care how good or bad you are” position. It would also seem to be inconsistent with the “develop your strengths” movement where people are encouraged to leverage their strengths (in a nutshell). The second point is sound advice regarding not using 360 results in isolation, but doesn’t give me much faith in the rigor of his 360 data.
If we are going to use 360 results to make decisions about employees, that means that someone is going to get more of something (e.g., pay, promotions, development opportunities, training, high potential designation) and someone is going to get less based, at least in part, on the 360 data. That is what “decision making” means.
Marc speaks of transparency in use of 360 as a central premise to his approach. If that is the case, we can start by being totally “transparent” with our raters by telling them how their ratings are being used. If it is a totally “development only” process, use a within-person (ipsative) scale with explicit directions to not compare the ratee to others when assigning ratings.
If the 360 is supporting decision making, ask the rater to help us make comparisons by using what I call a “normative” scale. I have successfully used normative scales, and they usually look something like this:
5 = Role Model (in the top 5-10%)
4 = Above Average (in the top 20-25%)
3 = Comparable to Other Leaders
2 = Below Average (in the bottom 20-25%)
1 = Far Below Average (in the bottom 5-10%)
The directions to the raters can help define the comparison group, or “other leaders.” But clearly we are creating a frame of reference for the rater that encourages something closer to a normal distribution and a direct attack on leniency error. I believe that the traditional leniency problem with 360 problems is at least partially attributable to the ambiguousness of common rating scales such as the Likert (Agree/Disagree) scale where the user (rater) is left to attach their own meaning to the scale points. Rater training can help combat rating errors, but, as I have noted before, is rarely implemented.
Want to be transparent and communicate the most important decision about your 360 process, i.e., its purpose? Put your scale where your money is (or isn’t): ipsative for development only, normative for decision making.
©2010 David W. Bracken
Subscribe to comments with RSS.