Here is what I think
I was working on my next blog topic when I happened across Ken Nowack’s latest offering (http://bit.ly/bQt2iP), titled “Another look in the mirror – does 360 feedback really work?” Ken and I have not really worked together but have communicated a number of times over the years on various 360 topics and initiatives. I always enjoy his perspectives and thoughtful approach to this and other topics in the HR realm. He was also nice enough to contact and congratulate me when I announced the start of this blog, at which time he also pointed me to a resource where some of his writing is published (www.results.envisialearning.com ) and offered to start up a discussion or debate as needed.
You will have to go to Ken’s blog to see what this is all about, of course, which is a list of “What would you think…” statements that cite some research sources that provide some thought-provoking results that are designed to question the utility (or even destructiveness) of 360 feedback. I am certainly not in any position to guess as to Ken’s motivation in creating this document, especially since he is a major practitioner in the field. I would say that, as a standalone article, it could be misused if taken out of context. I would hope that he is being a “devil’s advocate” and hoping to draw out people like me who might rise to the defense of the potential effectiveness of an effectively designed and implemented 360 system. So I will take the bait.
Here’s what I think:
I THINK that if a doctor offered me a drug where 33% of the patients get worse, I would find a new doctor. (I would also wonder what a “wonder drug” really is.)
I THINK that no two 360 processes are the same. There are over 100 design and implementation decisions that have to be made (or ignored), most of which having the potential to have a major impact on the effectiveness the system.
I THINK some 360 processes are poorly designed and do indeed have the potential to be destructive. Fortunately, they usually die a timely death. Unfortunately, the damage they leave is hard to undo.
I THINK that this list of research articles has a “half empty” feel to it.
I THINK that meta analyses are somewhat useful but limited by my first point, i.e., the major factors that can affect the success of a system (i.e, creation of behavior change), and that many of those major factors cannot be assessed in a meta analysis because either the studies don’t include them as variables or not enough studies do report them to make the findings reliable. A starter list of possible moderators might include: purpose of system, custom vs. standard instrument, length of instrument, rating scale, number of raters included, how raters are selected, rater training (a very important issue), manager approval (or not), technology, report format (e.g., level of detail), sharing of results with boss and/or raters, involvement of a coach, accountability, goal setting, relation to HR systems (e.g., succession planning, staffing, training, high potential program, performance management), and so on.
I THINK that some very good meta analyses, like the Smither et al one that Ken references, do begin to identify some process moderators that improve the probability of creating behavior change. To quote from that article, “We therefore think it is time for researchers and practitioners to ask “Under what conditions and for whom is multisource feedback likely to be beneficial?” (rather than asking “Does multisource feedback work?”).” (p.60). They point out that, indeed, 360 will not work for everyone. But they note many factors that have been shown to improve the probability of behavior change, many of which are in need of more research.
I THINK that expecting “large” levels of behavior change across large numbers of people is not realistic. Most organizations would be happy with a little change across a large number of leaders.
I THINK detecting behavior change is also a major challenge in terms of measurement. How long should/do we wait between measurements? What is measured, especially if/when the participants are asked to work on only 1-3 behaviors/skills (which will differ for each person)?
I THINK that the quality movement of the 80’s taught us that ratings of quality can go down as raters become “better” at the task. If/when we train raters and/or they learn more about the topic (quality or, in this context, leader behavior) they change their frame of reference. Although the actual raters may be the same people (by name) at time 1 and time 2, often they are not the “same” in other ways as they have changed their views of how a “good leader” is defined.
I KNOW that there are a number of studies that indicate that significant behavior change is observed with the major caveat that the participant involves the raters AFTER the data is back (Goldsmith and Underhill, 2001; Goldsmith and Morgan, 2004). That is one process design factor that too many organizations ignore or even actually discourage (or prohibit), a decision that suggests behavior change is not their primary objective.
I THINK that self ratings per se are pretty useless. Noting that they don’t relate to ratings by others is not very informative nor surprising. That said, the action of requiring self ratings helps by a) requiring the participant to be aware of the content, and b) demonstrating a commitment to the process, albeit at a minimal level.
I THINK that rater groups (managers, peers, direct reports, customers, etc.) not showing high agreement is why we do 360 and report them out separately in the results. I see that as a benefit of the process, not a drawback.
I THINK that the Watson Wyatt “study” is seriously flawed and discounted by most professionals. For starters, the causality that is inferred (that doing 360 harms financial performance) is irresponsible.
I THINK that the decisions that organizations make in the design and implementation of 360 processes treat the raters (short instruments) and the company (few raters) as the primary “customer” of the process when the participants should be the primary customers, needing reliable feedback data that is collected and used in a fair consistent way.
I THINK that asking “does 360 feedback really work” is much too simple and has too many qualifiers to be answered. Does a car “work” if it doesn’t have gas? If the driver is blindfolded? If he/she is 12 years old? Drunk? Driven at 150 mph? Doesn’t have seat belts? (this list could be as long as one for 360’s, i.e., over 100 items long, so I will spare you).
I THINK that readers of this blog who are interested in the many decisions that affect the success (validity) of a 360 system might like this article: “360 Feedback From Another Angle,” (Bracken, Timmreck, Fleenor and Summers, 2001). Contact me at email@example.com if you want a copy.
Bracken, D.W., Timmreck, C.W., Fleenor, J.W., & Summers, L. (2001). 360 degree feedback from another angle. Human Resource Management, 40 (1), 3-20.
Goldsmith, M., and Morgan, H. (2004). Leadership is a contact sport: The “follow-up” factor in management development. Strategy+Business, 36, 71-79.
Goldsmith, M., & Underhill, B.O. (2001). Multisource feeedback for executive development. In Bracken, D.W., Timmreck, C.W., and Church, A.H. The Handbook of Multisource Feedback. San Francisco: Jossey-Bass.
©2010 David W. Bracken