Strategic 360s

Making feedback matter

Posts Tagged ‘rater training

The Missing Link

leave a comment »

[tweetmeme source=”anotherangle360″]

My colleague, Jeff Saltzman, has a great blog that is much more diverse than mine ( ). His most recent entry begins with this little gem of a story that I want to plagiarize and take in my own direction:

”Which is more important, the sun or the moon?” A citizen of a small town not noted for its intellectual prowess asked. “Why the moon of course,” was the reply. “It shines at night when it is needed. The sun shines only during the day, when there is no need of it at all!” (Ausbel, N., A Treasury of Jewish Folklore, 1948)

I have touched on the topic of importance in some past blogs (, and the folly of asking raters what is “important”. This little story made me think of that issue once again from a slightly different angle. My stance has been, and still is, that raters are in a very poor position to judge the importance of a competency/behavior in the context of the need of the ratee and the organization.

There is really no way to know what is going through a rater’s mind if/when we ask him/her to give use importance ratings. There may be some research on this question (e.g., correlation between importance and effectiveness ratings), but I will hazard a guess that importance ratings are more a function of rater needs than the needs of the ratee or organization/team.

Jeff’s story also makes me wonder how qualified raters are to provide importance ratings when they are most likely not given any instruction as to what “importance” means (as rater training might attempt to do). And their rationale for importance ratings may well be as convoluted as the small town citizen’s is.

The question of importance is useful in helping prioritize actions. So, if it is not the raters who should indicate importance, who is it? The manager (“boss”), of course, partnering with the ratee. Hopefully the boss and ratee have a history of development discussions on a personal level, and about organization/team priorities to create alignment. If they have not been having those discussions, maybe a 360 process tied to performance management and development might create some mutual accountability for doing so.

The importance of the “boss” in the 360 process and employee development in general is so critical that it boggles the mind to think of 360’s that totally bypass (exclude) the manager. I will equally dismayed to read of a major 360 process describe on LinkedIn that makes boss input optional.  Really? I have always thought that manager input is the most useful feedback many ratees get out of 360’s, to the extent that a best practice is to require that the boss complete their input in order for a report to be generated.

I will go as far as to say the manager ratings are more important than participant self-ratings. Ideally both will happen but, as I mentioned in a recent blog, self- ratings are more an indication of commitment to the process than a true evaluation of self competence in many, many cases.  I will acknowledge that sometimes bosses use their ratings to send a message to the ratee, but even then the resulting discussion is often very enlightening for the ratee.

©2011 David W. Bracken

Built to Fail/Don’t Let Me Fail

leave a comment »

[tweetmeme source=”anotherangle360″]

This is a “two sided” blog entry, like those old 45 rpm records that had hit songs on both sides (think  “We Can Work It Out”/”Daytripper” by the Beatles),though my popularity may not be quite at their level.  This is precipitated by a recent blog (and LinkedIn discussion entry) coming from the Envisia people. The blog entry is called, “Does 360-degree feedback even work?” by Sandra Mashihi and can be found at   It would be helpful if you read it first, but not necessary.

Sandra begins by citing some useful research regarding the effectiveness of 360 processes. And she concludes that sometimes 360’s “work” and sometimes not.  Her quote is, “Obviously, the research demonstrates varied results in terms of its effectiveness.”

What is frustrating for some of us are the blanket statement about failures (and using terms like “obvious”) without acknowledging that many 360’s are “built to fail.” This is the main thesis of the article Dale Rose and I just published in the Journal of Business and Psychology.

Dale and I propose four features needed in a 360 process if it is likely to create sustainable behavior change:

1)      Reliable measurement: Professionally developed, custom designed instruments

2)      Credible data: Collecting input from trained, motivated raters with knowledge of ratees

3)      Accountability: Methods to motivate raters and ratees to fulfill their obligations

4)      Census participation: Requiring all leaders in an organizational unit to get feedback

We go on to cite research that demonstrates how the failure to build these features into 360 can, in some cases, almost guarantee failure and/or the ability to detect behavior change when it does occur. One such feature, for example, is whether the ratee follows up with raters (which I have mentioned in multiple prior blogs). If/when a 360 (or a collection of 360’s, such as in a meta analysis) is deemed a “failure”, I always want to know things such as whether raters were trained and whether follow up was required, for starters.

We are leaning more and more about the facets that increase the probability that behavior change will occur as a result of 360 feedback. Yet all too often these features are not built into many processes, and practitioners are surprised (“shocked, I’m shocked”) when it doesn’t produce desired results.

Sandra then goes on to state: “I have found 360-degree feedback worked best when the person being rated was open to the process, when the company communicated its purpose clearly, and used it for development purposes.” I assume that she means “development only” since all 360’s are developmental.  I definitely disagree with that feature. 360’s for “development (only) purposes” usually violate one or more of the 4 features Dale and I propose, particularly the accountability one. They often do not generate credible data because too few raters are used, even the best practice of including all direct reports.

The part about “being open to the process” is where I get the flip side of my record, i.e., don’t hurt my feelings.  In one (and only one) way, this makes sense. If the ratee doesn’t want to be in a development-only process, then by all means don’t force them. It is a waste of time and money. On the other hand, all development only processes are a waste of money in my opinion for most people. (And, by the way, development only is very rare if that means that no decisions are being made as a result.)

But if we ARE expecting to get some ROI (such as sustained behavior change) from our 360’s, then letting some people to opt out so their feelings aren’t hurt is totally contrary to helping the organization manage its leadership cadre. Intuitively, we should expect that those who opt out are the leaders that need it the most, who know that they are not effective and/or are afraid to be “discovered” as the bullies, jerks, and downright psychopaths that we know exist out there.

I have some fear that this fear of telling leaders that they are less than perfect is stemming from this troubling trend in our culture where everyone  has to succeed. I think that the whole “strengths” movement is a sign of that.

Over the last couple of weeks, I have seen a few things that further sensitized me to this phenomenon. One big one is this article in The Atlantic:  Protecting our children from failure is not working. Protecting our leaders from failure is also dooming your organization.

I swear I never watch America’s Funniest Videos, but during a rain delay of a baseball game recently, I did stumble upon it and succumbed. AFV is all about failure, and I’m not so sure that people always learn from these failures. But one video I enjoyed showing a 2 year old boy trying to pour apple juice from a BIG bottle into a cup. He put the cup on the floor and totally missed the first two times (with the corresponding huge mess). As a parent and grandparent, I was quite amazed that the person behind the camera just let it happen. But on the third try, the task was accomplished successfully, followed by applause and smiles! There was a huge amount of learning that occurred in just a minute or two because the adults allowed it to happen, with a bit of a mess to clean up.

How many of us would have just poured the juice for him? His learning isn’t over; he will make more mistakes and miss the cup occasionally. But don’t we all.

As a parting note, Dale and I support census participation for a number of reasons, one of which is the point I have already made about otherwise missing the leaders that need it most. We also see 360’s as a powerful tool for organizational change, and changing some leaders and not others does not support that objective. Having all leaders participate is tangible evidence that the process has organization support and is valued. Finally, it creates a level playing field for all leaders for both evaluation and development, communicating to ALL employees what the organization expects from its leaders.

©2011 David W. Bracken

Has Anything Changed in 10 Years?

leave a comment »

[tweetmeme source=”anotherangle360″]

2011 marks the 10th anniversary of the publication of The Handbook of Multisource Feedback. To mark this occasion, we have convened a panel of contributors to The Handbook for a SIOP (Society of Industrial and Organizational Psychology) session to discuss how the field of 360 has changed (and not changed) in those 10 years. Panel members will include the Editors (Carol Timmreck (who will be moderator), Allan Church and myself), James Farr, Manny London, David Peterson, Bob Jako, and Janine Waclawksi. (See for more information.)

In a “good news/bad news” kind of way, we frequently get feedback from practitioners who still use The Handbook as a reference. In that way, it seems to be holding up well (the good news). The “bad news” might be that not much has changed in 10 years and the field is not moving forward.

Maybe the most obvious changes have been in the area of technology, again for good and bad. One of the many debates in this field is whether putting 360 technology in the hands of inexperienced users really is such a great idea. That said, it is a fact that it is happening and will have some potential benefits in cost and responsiveness.

Besides technology, what how else has the field of 360 feedback progressed or digressed in the last decade?

I will get the ball rolling by offering two pet peeves:

1)      The lack of advancement in development and use of rater training as a best practice, and

2)      The ongoing application of a testing mindset to 360 processes.

Your thoughts?

©2011 David W. Bracken

The Death Card

leave a comment »

[tweetmeme source=”anotherangle360″]

A number of (pre-recession) years ago, I belonged to a firm that was operating in the black and held some very nice off-site meetings for its consultants. At one such event, we had an evening reception that had some fun activities, one of which being a Tarot reader. I don’t even read horoscopes but there was no one waiting and I decided to give it a try (the first and last time).  I obviously didn’t know much about Tarot but it seemed like the last card to be turned over was the most important. And, lo and behold, it was the Death card! I remember a pause from the Reader (perhaps an intake of breath?), and then a rapid clearing of the cards with some comment to the effect of, “That’s not important.”  Session over.

Well, I guess the good news is that I am still here (most people would agree with that I think).  My purpose for bringing this up is not to discuss superstitions and the occult, but to reflect on how people react to and use 360 feedback.

In fact, I have been known to call some 360 processes “parlor games, “which relates directly to my Tarot experience. That was a true “parlor game.”  What is a parlor game? My definition, for this context, is an activity that is fun and has no consequences, where a person can be the focus of attention with low risk of embarrassment and effort.  Since I strongly believe in self determination, I do my best to not let arbitrary events that I cannot control to affect my life. That would include a turn of a card, for starters.

So how do we ensure that 360 Feedback isn’t a parlor game and does matter? I propose that two important factors are Acceptance and Accountability.

Some of the design factors that promote Acceptance would include:

  • Use a custom instrument (to create relevance)
  • Have the rater select raters, with manager approval (to enhance credibility of feedback)
  • Enhance rater honesty and reliability (to help credibility of data)
  • Invite enough raters to enhance reliability and minimize effects of outliers
  • Be totally transparent to purpose, goals, and use (not mystical, magic, inconsistent or arbitrary)

Factors that can help create Accountability (and increase the probability of behavior change) include:

  • Require leaders to discuss results and development plans with raters (like going public with a New Year’s Resolution)
  • Include results as a component of performance management, typically in the development planning section, to create consequences for follow through, or lack thereof
  • Ensure that the leader’s manager is also held accountable for properly using results in managing and coaching
  • Conduct follow-up measures such as mini-360’s and/or annual readministrations.

Some 360 processes appear to define success as just creating awareness in the participants, hoping that the leader will be self motivated to change. That does happen; some leaders do change, at least for a while, and maybe even in the right way. (Some people probably change based on Tarot readings too!).  For those leaders who need to change the most, it usually doesn’t happen without Acceptance and Accountability.

Simply giving a feedback report to a leader and stopping there seems like a parlor game to me. A very expensive one.

©2011 David W. Bracken

Maybe Purpose Doesn’t Matter?

leave a comment »

[tweetmeme source=”anotherangle360″]

While there are many discussions and debates within the 360 Feedback community (including one regarding randomizing items currently on LinkedIn that I will address in a later blog), probably none is more intense and enduring than the issue of the proper use of 360 results. In The Handbook of MultiSource Feedback, a whole chapter (by Manny London) was dedicated to “The Great Debate” regarding using 360 for developmental vs. decision making purposes. In fact, in the late 90’s an entire book was published by the Center for Creative Leadership based on a debate I organized at SIOP.

I have argued in earlier blogs and other forums that I believe this “either/or” choice is a false one for many reasons. For example, even “development only” uses require decisions that affect personal and organizational outcomes and resources. Also, even when used for decision (including succession planning, staffing, promotions, and, yes, performance management), there is always a development component.

One of the aggravating blanket statements that is used by the “development only” crowd is that respondents will not be honest if they believe that the results will be used to make decisions that might be detrimental to the ratee, resulting in inflated scores with less variability. I would say that, in fact, that is by far the most common argument for the “development only” proponents, and one that is indeed supported by some research studies.

I have just become aware of an article published 3 years ago in the Journal of Business and Psychology (JBP) relating to multisource feedback, titled “Factors Influencing Employee Intentions to Provide Honest Upward Feedback Ratings” (Smith and Fortunato, 2008).  For those of you who are not familiar with JBP, it is a refereed journal of high quality that should be on your radar and, in full disclosure, a journal for which I am an occasional reviewer.

The study was conducted at a behavioral health center with a final sample of 203 respondents. The employees filled out a questionnaire about various aspects of an upward feedback process that was being implemented in the future.

The article is fairly technical and targeted toward the industrial/organizational community. I have pulled out one figure for the geeks in the audience to consume if desired (click on “360 Figure”) . But let me summarize the findings of the study.

The outcome (dependent variable) that was of primary interest to the researchers is foreshadowed in the title, i.e., what factors lead to intentions to respond honestly in ratings of a supervisor (upward feedback).  The most surprising result (as highlighted in the discussion by the authors) was that purpose (administrative versus developmental) had no predictive value at all! Of all the predictor variables measured, it was the least influential with no practical (statistical) significance.

What does predict intentions to provide honest feedback? One major predictor is the level of cynacism, with (as you might guess) cynical attitudes resulting in less honesty. The study suggests that cynical employees fear retaliation by supervisors and are less likely to believe that the stated purpose will be followed. The authors suggest that support and visible participation by senior leaders might help reduce these negative attitudes. We also need to continue to protect both real and perceived confidentiality, and to have processes to identify cases of retaliation and hold the offending parties accountable.

The other major factor is what I would label as rater self confidence in their ability as a feedback provider. Raters need to feel that their input is appropriate and valued, and that they know how the process will work. They also have a need to feel that they have sufficient opportunity to observe.  The authors appropriately point to the usefulness of rater training to help accomplish these outcomes. They do not mention the rater selection process as being an important determinant of opportunity to observe, but that is obviously a major factor in ensuring that the best raters are chosen.

One suggestion the authors make (seemingly out of context) that is purported to help improve the honesty of the feedback is to use reverse-worded items to keep raters from choosing only socially desirable responses (e.g., Strongly Agree).  I totally disagree with practices such as reverse wording and randomization which may actually reduce the reliability of the instrument (unless the purpose is for research only). For example, at our SIOP Workshop, Carol Jenkins and I will be showing an actual 360 report that uses both of those methods (reverse wording and randomization). In this report (that Carol had to try to interpret for a client), the manager (“boss”) of the ratee had give the same response (Agree) to two versions of the same item where one was reverse scored. In other words, the Manager was Agreeing that the ratee was both doing and not doing the same thing.

Now what? The authors of this study seem to suggest that situations like this would invalidate the input of this manager, arguably the most important rater of all.  Now we could just contact the manager and try to clarify his/her input. But the only reason we know of this situation is that the manager is not anonymous (and they know that going into the rating process). If this same problem of rating inconsistency occurs with other rater groups, it is almost impossible to rectify since the raters are anonymous and confidential (hopefully).

This is only one study, though a well designed and analyzed study in a respected journal. I will not say that this study proves that purpose does not have an effect on honesty. Nor should anyone say that other studies prove that purpose does affect honesty. To be clear, I have always said that it may be appropriate to use 360 results in decision making under the right conditions, conditions that are admittedly often difficult to achieve. This is in contrast to some practitioners who contend that it is never appropriate to do so, under any conditions.

Someday when I address the subject of organizational readiness, I will recall the survey used in this research which was administered in anticipation of implementing an upward feedback process. This brief (31 item) survey used for this study would be a great tool to assess readiness in all 360 systems.

One contribution of this research is to point out that intention to be honest is as much a characteristic of the process as it is of the person. Honesty is a changeable behavior in this context through training, communication, and practice. Making blanket statements about rater behavior and how a 360 program should or shouldn’t be used are not productive.

360 Figure

©2011 David W. Bracken

Not Funny

leave a comment »

[tweetmeme source=”anotherangle360″]

I seem to be in a bit of a rut with themes around humor and now commercials. Despite trying to bypass as many commercials as possible with my DVR, occasionally I do see one and sometimes even for the better.

One that caught my eye/ear is one by IBM that starts with a snippet of a Groucho Marx (whom I also like very much) where he states, “This morning I shot an elephant in my pajamas.”  Of course, the fun part is when he follows, “How he got in my pajamas, I will never know.”  Ba bump.

The commercial goes on to talk about a computer called Watson that has been developed by IBM with capabilities that will be used to compete on Jeopardy (another favorite). The point is that language has subtle meanings, euphemisms, metaphors, nuances and unexpected twists that are difficult for machines to correctly comprehend.

In the context of 360 Feedback, the problem is that we humans are sometimes not so good at picking up the subtleties of language as well. We need to do everything we can to remove ambiguity in our survey content, acknowledging that we can never be 100% successful.

We have all learned, sometimes the hard way, about how our attempts to communicate with others. How often have we had to come to grips with how our seemingly clear directions have been misunderstood by others?

I became sensitized to this question of ambiguity in language during the quality movement of the 80’s and the work of Peter Senge as embodied in The Fifth Discipline and the accompanying Fifth Discipline Fieldbook. (Writing this blog has spurred me to pull out this book; if you youngsters are not aware of Senge’s writings, it is still worth digging out. There is a 2006 Edition which I confess I have not read yet.)

There are many lessons in these books regarding the need to raise awareness about our natural tendencies as humans to fall back on assumptions, beliefs, values, etc., often unconsciously, in making decisions, trying to influence, and taking actions. One lesson that has particularly stuck with me in the context of 360’s is the concept of mental models, which Senge defines as, “deeply ingrained assumptions, generalizations, or even pictures or images that influence how we understand the world and how we take action.”  In the Fieldbook, he uses an example of the word “chair” and how that simple word will conjure up vastly different mental images of what a “chair” is, from very austere, simple seats to very lush, padded recliners and beyond. (In fact, it might even create an image of someone running a meeting if we are to take it even farther.)

So Groucho created a “mental model” (or assumed one) of us visualizing him in his pajamas with a gun chasing an elephant. Then he smashes that “assumption” we made by telling us that the elephant was wearing the pajamas. That is funny in many ways.

Sometimes we are amused when we find we have made an incorrect assumption about what someone has told us. I have told the story before of the leader who made assumptions about his low score on “Listens Effectively.” He unexpectedly found that his assumptions were unfounded and the raters were simply telling him to put down his PDA. That could be amusing and also a relief since it is an easy thing to act on.

360 Feedback is a very artificial form of communication where we rely on questionnaires to allow raters to “tell” the ratee something while protecting their anonymity. This also has the potential benefit of allowing us to easily quantify the responses which, in turn, can be used to measure gaps (between rater groups, for example) and track progress over time.

Of course this artificial communication creates many opportunities for raters to misunderstand or honestly misuse the intent of the items and, in turn, for ratees to misinterpret the intended message from the raters. We need to do our best to keep language simple and direct, though we can never prevent raters applying different “mental models.”

Take an item like, “Ensures the team has adequate resources.” Not a bad question. But, like “chair,” “resources” can create all sorts of mental images such as people (staff), money (budget), equipment (e.g., computers), access to the leader, and who knows what else! We could create a different item for each type of resource if we had an unlimited item budget, which we don’t.

This potential problem is heightened if there will be multiple languages used, creating all sorts of issues with translations, cultural perspectives, language nuances, and so on.

In the spirit of “every problem has a solution,” I can think of at least four basic recommendations.

First, be diligent in item writing to keep confusion to a minimum.  For example:

  • Use simple words/language
  • Don’t use euphemisms (“does a good job”)
  • Don’t use metaphors (“thinks outside the box”)
  • Don’t use sports language (“creates benchstrength”)
  • Keep all wording positive (or cluster negatively phrased items such as derailers in one dimension with clear instructions)

Second, conduct pilot tests with live raters who can give the facilitator immediate feedback on wording in terms of clarity and inferred meaning.

Third, conduct rater training. Some companies tell me that certain language is “ingrained” in their culture, such as “think outside the box.” (I really wonder how many people really know the origins of that metaphor. Look it up in Wikipedia if you don’t.)  I usually have to defer to their wishes, but still believe that their beliefs may be more aspirational than factual. Including a review of company-specific language (which does have some value in demonstrating the uniqueness of the 360 content) during rater training will have multiple benefits.

Fourth, acknowledge and communicate that it is impossible to prevent misinterpretations by the senders (raters) and the receivers (ratees). This will require that the ratee discuss results with the raters and ensure that they are all “on the same page”. (metaphor intended with tongue in cheek).

I bet that some ratees do actually laugh (or at least chuckle) if/when they hear how some raters interpret the questions.  But more typically it is not funny. And it is REALLY not funny if the ratee invests time and effort (and organizational resources) taking action on false issues due to miscommunication.

(Note: For those interested, Carol Jenkins and I will be talking about these issues in our SIOP Pre-Conference workshop on 360 Feedback on April 13 in Chicago.)

©2011 David W. Bracken

There Are “Right” Answers

with 2 comments

[tweetmeme source=”anotherangle360″]

For those of you who might attend the next SIOP (Society for Industrial and Organizational Psychology) Conference in Chicago in April, I am pleased to note that we have been accepted to conduct a panel consisting of contributors to The Handbook of Multisource Feedback, which is approaching its 10th anniversary of publication. The panel is titled, “How has 360 degree Feedback evolved over the last 10 years?”  Panel members include Allan Church, Carol Timmreck, Janine Waclawski, David Peterson, James Farr, Manny London, Bob Jako and myself.

We received a number of thoughtful, useful comments and suggestions from the reviewers of the proposal, one of which stated this:

I would like to see a serious discussion of whether or not 360 is a singular practice. It seems as though 360 can be used with so many different interventions (succession, development, training needs analysis, supplement to coaching, …the list is HUGE) that when we say something like “is 360 legal” it is almost impossible to answer without many caveats regarding the details of the specific 360 process that was used. It’s almost as though we need to move on from ‘is 360 xyz’ to ‘if we do 360 this way, we get these outcomes and if we do 360 that way we get those outcomes.’ Can’t wait to hear the panel, this is much needed.

This is an extremely insightful observation. I have broached this topic in earlier blogs regarding alignment of purpose and decisions in design and implementation.  But there are some things that are required regardless of purpose.

To look at extremes, we might consider 360 processes where the N=1, i.e., where a single leader is given the opportunity to get developmental feedback. This is often in preparation for an experience such as a leadership development/training program, or some development program (e.g., high potential).  In these instances, it is an ad hoc process where an off-the-shelf instrument may be most practical. The instrument can be lengthy since raters will only have to fill it out one time. And typically there are major resources available to the participant in the form of coaches, trainers, and/or HR partners to ensure that the feedback is interpreted and used productively.

Compare the N=1 scenario to the N>1 process. By N>1, I use shorthand to indicate 360 processes that are applied across some segment of the population, such as a function, department, or entire organization. In these cases, it becomes much more important to have a custom designed instrument that reflects unique organization requirements (competencies, behaviors) that can create system change while simultaneously defining effective leadership to raters and ratees alike. The process requires some efficiencies due to many raters being involved, and some being asked to complete multiple forms.  We also need to plan for ways to support the many ratees in their use of the feedback.

BUT, we might also say that there are some things that are so basic as to be necessary whether N=1 or N>1.  Just this week I was sent this interview of Cindy McCauley (of the Center for Creative Leadership) ( Many readers will already know who Cindy is; if not, suffice to say she is highly respected in our field and has deep expertise in 360 Feedback. (In fact, she contributed a chapter to the book, “Should 360 Feedback Be Used Only for Development Purposes?” that I was also involved with.) In this interview, Cindy makes some important points about basic requirements for reliability and validity that I interpret to be applicable to all 360 processes.

What really caught my attention was this statement by Cindy:

…the scores the managers receive back mean a lot to them. They take them very seriously and are asked

to make decisions and development plans based on those scores. So you want to be sure that you can

rely on those scores, that they’re consistent and reflect some kind of accuracy.

I take the liberty (which Cindy would probably not) to expand the “make decisions” part of this statement to apply more broadly, that others (such as the leader’s manager) also use the feedback to make decisions. When she says that managers make decisions on their feedback, what decisions can they make without the support of the organization (in the person of their boss, most typically)? This is basically the crux of my argument that there is no such thing as “development only” processes. Development requires decisions and the commitment of organization resources. This only reinforces her point about the importance of validity and reliable measurement.

So what’s my point? My point is that I believe that too many ad hoc (N=1) 360 processes fall short of meeting these requirements for validity and reliability. Another debate for another time is whether off-the-shelf instruments have sufficient validity to measure unique organization requirements.  I do believe it is accurate to say that reliable measurement is often neglected in ad hoc processes when decisions are made about number of raters and quality of ratings.

For example, research indicates that raters have different “agendas” and that subordinates are the least reliable feedback providers, followed by peers and then managers. Lack of reliability can be combated in at least two ways: rater training and number of raters. We can put aside rater training (beyond having good instructions); it rarely happens despite its power and utility.

So we can improve reliability with numbers. In fact, this is really why 360 data is superior to traditional, single source evaluations (i.e., performance appraisals).  For N>1 processes, I STRONGLY recommend that all direct reports (subordinates) participate as raters. This has multiple benefits, including beefing up the number of raters for the most unreliable rater group. Then, for peers, aiming for 5-7 respondents is recommended.

My contention is that the majority of ad hoc (N=1) processes do not adhere to those guidelines. (I have no data to support that assertion, just observation.)  The problem of unreliable data due to inadequate number of raters is compounded by the fact that the decisions resulting from that flawed data are magnified due to the senior level of the leaders and the considerable organization resources devoted to their development.

When I started writing this blog, I was thinking of the title, “There is No “Right Answer,” meaning that decisions need to fit the purpose. But actually there are some “Right Answers” that apply regardless of purpose. Don’t let the “development only” argument lead to implementation decisions that reduce the reliability and validity of the feedback. In fact, many guidelines should apply to all 360 processes, whether N=1 or N>1.

©2011 David W. Bracken

It’s wonderful, Dave, but…

with 2 comments

[tweetmeme source=”anotherangle360″]

This is one of my favorite cartoons (I hope I haven’t broken too many laws by using it here; I’m certainly not using it for profit!).  I sometimes use it to ask whether people are more “every problem has a solution” or “every solution has a problem” types. Clearly, Tom’s assistant is the latter.

I thought of this cartoon again this past week during another fun (for me, at least) debate on LinkedIn about the purpose of 360’s, primarily about the old decision making vs. development only debate.

Now, I don’t believe that 360 is comparable to the invention of the light bulb (though there is a metaphor lurking in there somewhere), nor did I invent 360. But, as a leading proponent of using 360 for decision making purposes (under the right conditions), by far the most common retort is something along the lines of, “It’s (360) wonderful, Dave, but using it for decisions distorts the responses when raters know it might affect the ratee.”

Yes, there is some data that suggests that raters report their ratings would be affected if they knew they would penalize the ratee in some way.  And it does make intuitive sense to some degree. But I offer up these counterpoints for your consideration:

  • I don’t believe I have ever read a study (including meta analyses) that even considers, let alone studies, rater training effects, starting with whether it is included as part of the 360 system(s) in question. In my recent webinar (Make Your 360 Matter), I presented what I think is some compelling data from a large sample of leaders on the effects of rater training and scale on 360 rating distributions. (We will discuss this data again at our SIOP Pre-Conference Workshop in April.) In the spirit of “every problem has a solution,” I propose that rater training has the potential to ameliorate leniency errors.
  • There is a flip side to believing that your ratings will affect the ratee in some way, which, of course, is believing that your feedback doesn’t matter. I am not aware of any studies that directly address that question, but there is anecdotal and indirect evidence that this also has negative outcomes. What would you do if you thought your efforts made no difference (including not being read)? Would you even bother to respond? Or take time to read the items? Or offer write in comments? Where is the evidence that “development only” data is more “valid” than that used for other purposes?  It may be different, but that does not always mean better.

The indirect data I have in mind are the studies published by Marshall Goldsmith and associates on the effect of follow up on reported behavioral change. (One chapter is in The Handbook of MultiSource Feedback; another article is “Leadership is a Contact Sport,” which you can find at  The connection I am making here is in suggesting that lack of follow up by the ratee can be a signal that the feedback does not matter, with the replicated finding that reported behavior change is typically zero or even negative. Conversely, when the feedback does matter (i.e., the ratee follows up with raters), behavior change is almost universally positive (and increases with the more follow up reported).

It’s all too easy to be an “every solution has a problem” person. We all do it. I do it too often. But maybe it would help if we became a little more aware of when we are falling into that mode.  It may sound naïve to propose that “every problem has a solution,” but it seems like a better place to start.

©2010 David W. Bracken

I see you rolling your eyes

with 2 comments

[tweetmeme source=”anotherangle360″]

This is the second of a series of blog entries that I am using to respond to some questions that were submitted during my recent webinar, “Make Your 360 Matter.”  Some of these questions were ones I got to during the webinar, but would like to expand on my answer and at the same time share the thoughts with others who might not have attended.

I am going to combine two questions to address one important topic:

You went over this Dave, but if you could pick 1-3 things that most 360s do poorly–and could easily do better

­Rater training seems so important, but it’s hard enough to get ANY training out there.  Are there streamlined ways to get the key points conveyed to raters?

For ten years or more, I (and some coauthors) have contended that rater training might well be the most important and neglected practice in 360 systems. My suspicions are that at least part of the reason for this lies in the tone of the second question, i.e., the “rolling of the eyes” every time the topic of rater training comes up due to preconceptions of what it involves.

During the webinar, I presented data on two 360 practices that appear to have major effects on the ability to create and measure behavior change. One of those design features is follow up with raters with data from the article, “Leadership is a Contact Sport,” which can be found on

The second topic was actually a combination of practices, i.e., choice of rating scale combined with rater training. I presented some data that strongly indicates the potential power of those factors in affecting the distribution of ratings, especially in reducing leniency error.

If we think about it a little, every 360 process already (hopefully) has some sort of rater training, usually in the form of directions on how to complete the questionnaire. In its simplest form, that might consist only of basic directions on how to physically complete the survey (e.g., one answer per question, must answer every question).  I find it is becoming increasingly common for instructions to go a little further by providing further guidance to the rater, such as:

  • Think of the leader’s behavior during the past year
  • Do not give excessive weight to recent events or observations
  • Use the full rating scale as appropriate. No leader is so good to deserve all “5’s” or so bad to get all “1’s”

We also often give a form of “training” in regard to write in comments:

  • Be specific about what you observed and/or what you suggest
  • Be constructive
  • Limit you r comments to job-related behaviors
  • Do not identify yourself, unless you intentionally desire to do so

In recent blogs and the webinar as well, I have expressed the view that 360 surveys are not “tests.” In the classic view of tests, we strive to identify individual differences that will help us differentiate subjects in order to predict future behavior, such as success on the job. In 360’s, we have no desire or need to measure individual differences in the raters since they are not the focus of our measurement efforts. Instead, we use training to minimize or remove individual differences in raters that we define as rater error.

So how do we deliver rater training? I have seen it come in many forms, including classroom training. But its most common design is a set of slides that the rater must, at a minimum, review prior to completing the first questionnaire. Once they have done that, they are “certified” and do not have to participate in any other training for that administration cycle.

Some of the typical content areas for rater training can include:

  • Purpose of the 360 process
  • How the feedback will be used
  • How anonymity is protected
  • Source of the behavioral items (e.g., values, competency model)
  • Rating scale format/content
  • When to respond “Not Observed” (Don’t Know)
  • Time frame (e.g., 1 year of behavior)
  • Types of rating errors and how to avoid (e.g., leniency, severity, recency, halo)
  • Case studies/examples (e.g., how to use rating scale)

There is a lot of variability in both amount of content and the pace that raters go through the slides, so the time required is hard to predict. Some processes have some sort of test at the end to ensure some level of attention.

Don’t discount using group training if the setting allows it. I once did a 360 at a hospital and was able to convene groups of nurses for 20-30 minute sessions, for example. Based on their questions, I know that it improved the quality of the feedback by correcting misconceptions and/or misinformation.

Another observation I made in the webinar is that rater training is a best practice in performance management/appraisal processes and a guard against legal challenge. Since 360’s closely resemble performance appraisals, this would seem applicable to them as well.

So please stop rolling your eyes and consider the potential benefit, with little cost, in implementing some form of rater training. It can make a difference.

Please share any experience you have had with rater training, good or bad!!

©2010 David W. Bracken