Strategic 360s

Making feedback matter

Frequency: Too Often

with 4 comments

[tweetmeme source=”anotherangle360″]

I delivered a webinar last week on using 360 Feedback in Performance Management Processes (PMP), partially built upon a recent article that Allan Church and I published in HRPS’s People & Strategy journal on that topic (let me know if you want a copy).  In the webinar, I spent a little time talking about the challenges of creating reliable/valid measurement when we are relying on input not from the target person but from observers of his/her behavior. 

One of the many elements that come into play when asking employees to rate something (a person, an organization) is the rating scale that is being used.  Note also that the rating scale’s effectiveness is likely to be directly affected by the quality of rater training, which is often neglected beyond the most basic of written instructions. 

In the webinar, I shared a list of a dozen or so various rating scales that I have encountered over the years, all in a 5 point format.  We also see in The 3D Groups recent benchmark study of over 200 organizations that use 360 feedback that, by far, the 5 point scale and the Likert Agree/Disagree format are used more often than any other scale type.  I’m not going too far out on a limb to propose that the use of the 5 point Likert scale is a carry over from employee surveys.  While there is something to be said for familiarity, I also propose that this practice is a form of laziness in 360 designers who haven’t reflected long or hard enough to consider scales that work better when the target is a specific person and not some nebulous entity like an organization that is the focus of the engagement survey.

I have advocated for the need to have the scale to match the purpose in an earlier blog (https://dwbracken.wordpress.com/2010/09/02/put-your-scale-where-your-money-is-or-isnt/) so I will move on to another pet peeve.

In the last few weeks, I pulled together a group of colleagues to submit a proposal for a SIOP symposium on helping managers to be better coaches. This process is always fun when you see research others are conducting in an area where you have special interests (kind of like buying a box set of CDs by a favorite artist and discovering some less well known gems).  One of the research papers demonstrates once again the inadequacy of frequency scales (typically 5 point scales that ask how often the person does something, ranging from Never to Always). 

Frequency scales continue to be widely used.  The aforementioned 3D Group study indicates that 23% of the reporting organizations use this scale, third most often behind Agreement (49%) and Effectiveness (31%) (which adds up to more than 100%; it may be that companies were allowed to report on more than one 360 process in their organization).  Frankly, the 23% is shockingly high.  Very recently (https://dwbracken.wordpress.com/2013/08/11/what-is-a-coach-redux/) I cited a study that presents a newly developed questionnaire about manager behaviors in the context of performance management that uses a frequency scale, to my chagrin.

For starters, a frequency scale is conceptually flawed. People can’t do everything “Always” (or even Almost Always, as some scales use).  And because they do something “always” doesn’t mean they do it well, and, conversely, because they do it Rarely or Never doesn’t mean they are bad at it. 

As importantly, every time I have seen them scrutinized in research, frequency scales come out poorly in comparison to other formats in terms of reliability and validity.  This is the 20th anniversary of a paper Karen Paul (now at 3M) and I presented at SIOP that indicated that frequency scales severely penalize supervisors who do some things infrequently but are otherwise perceived to be effective.

In a (frankly) more rigorous piece of research by Kaiser and Kaplan (2006) (that you can access here: http://kaplandevries.com/thought-leadership/list/C44), they also demonstrate that frequency scales are, by far, less satisfactory when compared to Evaluative and “Do More/Do Less” scales.

Frequency scales are used far too frequently.  They should be used Never.

 

Kaiser, R.B., & Kaplan, R.E. (2006, April). Are all scales created equal? Response format and the validity of managerial ratings. Paper in B.C. Hayes (Chair), The Four “Rs” of 360º Feedback: Second Generation Research on Determinants of Its Effectiveness, symposium presented at the 21st Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, TX.

 

©2013 David W. Bracken

Advertisements

Written by David Bracken

September 25, 2013 at 1:09 pm

4 Responses

Subscribe to comments with RSS.

  1. Hi Dave,

    Hope you are doing well – I continue to appreciate your posts and wish they were even more… frequent! I would love a copy of the article you and Allan published…

    Adam

    Carroll, Adam (NYC-IPG)

    September 25, 2013 at 1:18 pm

    • Thanks, Adam! I will send the article shortly. Hope you find it interesting and useful! Dave

      David Bracken

      September 25, 2013 at 1:39 pm

  2. I’m glad I found my way to your post that helps clarify the issues with ratings, coupled with little or non-existent instruction, misinterpretation, etc.

    I’ve found, similar to your ending comments, that a simple scale with description: Do more of, Stay the Same, Do Less of” with perhaps three to five items takes a group further, especially in “tender” groups, those who may have limited trust and openness. In such groups, making the “do less of” voluntary, helps to some degree, particularly if it is seen as an honest option, with no pressure to participate in “improvement” feedback.

    Thanks again for the post. I’, sharing excerpts of it on my Talent and Performance Development ScoopIt curation stream.

    Deb Nystrom

    December 1, 2013 at 3:41 pm


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: