Frequency: Too Often
I delivered a webinar last week on using 360 Feedback in Performance Management Processes (PMP), partially built upon a recent article that Allan Church and I published in HRPS’s People & Strategy journal on that topic (let me know if you want a copy). In the webinar, I spent a little time talking about the challenges of creating reliable/valid measurement when we are relying on input not from the target person but from observers of his/her behavior.
One of the many elements that come into play when asking employees to rate something (a person, an organization) is the rating scale that is being used. Note also that the rating scale’s effectiveness is likely to be directly affected by the quality of rater training, which is often neglected beyond the most basic of written instructions.
In the webinar, I shared a list of a dozen or so various rating scales that I have encountered over the years, all in a 5 point format. We also see in The 3D Groups recent benchmark study of over 200 organizations that use 360 feedback that, by far, the 5 point scale and the Likert Agree/Disagree format are used more often than any other scale type. I’m not going too far out on a limb to propose that the use of the 5 point Likert scale is a carry over from employee surveys. While there is something to be said for familiarity, I also propose that this practice is a form of laziness in 360 designers who haven’t reflected long or hard enough to consider scales that work better when the target is a specific person and not some nebulous entity like an organization that is the focus of the engagement survey.
I have advocated for the need to have the scale to match the purpose in an earlier blog (https://dwbracken.wordpress.com/2010/09/02/put-your-scale-where-your-money-is-or-isnt/) so I will move on to another pet peeve.
In the last few weeks, I pulled together a group of colleagues to submit a proposal for a SIOP symposium on helping managers to be better coaches. This process is always fun when you see research others are conducting in an area where you have special interests (kind of like buying a box set of CDs by a favorite artist and discovering some less well known gems). One of the research papers demonstrates once again the inadequacy of frequency scales (typically 5 point scales that ask how often the person does something, ranging from Never to Always).
Frequency scales continue to be widely used. The aforementioned 3D Group study indicates that 23% of the reporting organizations use this scale, third most often behind Agreement (49%) and Effectiveness (31%) (which adds up to more than 100%; it may be that companies were allowed to report on more than one 360 process in their organization). Frankly, the 23% is shockingly high. Very recently (https://dwbracken.wordpress.com/2013/08/11/what-is-a-coach-redux/) I cited a study that presents a newly developed questionnaire about manager behaviors in the context of performance management that uses a frequency scale, to my chagrin.
For starters, a frequency scale is conceptually flawed. People can’t do everything “Always” (or even Almost Always, as some scales use). And because they do something “always” doesn’t mean they do it well, and, conversely, because they do it Rarely or Never doesn’t mean they are bad at it.
As importantly, every time I have seen them scrutinized in research, frequency scales come out poorly in comparison to other formats in terms of reliability and validity. This is the 20th anniversary of a paper Karen Paul (now at 3M) and I presented at SIOP that indicated that frequency scales severely penalize supervisors who do some things infrequently but are otherwise perceived to be effective.
In a (frankly) more rigorous piece of research by Kaiser and Kaplan (2006) (that you can access here: http://kaplandevries.com/thought-leadership/list/C44), they also demonstrate that frequency scales are, by far, less satisfactory when compared to Evaluative and “Do More/Do Less” scales.
Frequency scales are used far too frequently. They should be used Never.
Kaiser, R.B., & Kaplan, R.E. (2006, April). Are all scales created equal? Response format and the validity of managerial ratings. Paper in B.C. Hayes (Chair), The Four “Rs” of 360º Feedback: Second Generation Research on Determinants of Its Effectiveness, symposium presented at the 21st Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, TX.
©2013 David W. Bracken
Subscribe to comments with RSS.