Being the 360 Feedback nerd I am, I love it when some new folks get active on the LinkedIn 360 discussion group. One discussion emerged recently that caught my eye, and I have been watching it with interest, mulling over the perspectives and knowing I had to get my two cents in at some point.

Here is the question:

How many raters are too many raters?

We normally recommend 20 as a soft limit. With too many, we find the feedback gets diluted and you have too many people that don’t work closely enough with you to provide good feedback. I’d be curious if there are any suggestions for exceptions.

This is an important decision amongst the dozens that need to be made in the course of designing and implementing 360 processes. The question motivated me to pull out The Handbook of Multisource Feedback and find the excellent chapter on this topic by James Farr and Daniel Newman (2001), which reminded me of the complexity of this decision. Let me also reiterate that this is another decision that has different implications for “N=1” 360 processes (i.e., feedback for a single leader on an ad hoc basis) versus “N>1” systems (i.e., feedback for a group of participants); this blog and discussion is focused on the latter.

Usually people argue that too many surveys will cause disruption in the organization and unnecessary “soft costs” (i.e., time). The author of this question poses a different argument for limiting the rater population, which he calls “dilution” due to inviting unknowledgeable raters.  For me, one of the givens of any 360 system is that the raters must have sufficient experience with the ratee to give reliable feedback. One operationalization of that concept is to require that an employee must have worked with/for the ratee for some minimum amount of time (e.g., 6 months or even 1 year), even if he/she is a direct report. Having the ratee select the raters (with manager approval) is another practice that is designed to help get quality raters that then also facilitate the acceptance of the feedback by the ratee. So “dilution” due to unfamiliarity can be combated with that requirement, at least to some extent.

One respondent to this question offers this perspective:

The number of raters depends on the number of people that deal with this individual through important business interactions and can pass valuable feedback based on real experience. There is no one set answer.

I agree with that statement. Though, while there is no one set answer, some answers are better than others (see below).

In contrast, someone else states:

We have found effective to use minimum 3 and maximum 5 for any one rater category.

The minimum of 3 is standard practice these days as a “necessary but not sufficient” answer to the number of raters. As for the maximum of 5, this is also not uncommon but seems to ignore the science that supports larger numbers.  When clients seek my advice on this question of number of raters, I am swayed by the research published by Greguras and Robie (1998) who collected and researched the question of the reliability of various rater sources (i.e., subordinates, peers and managers). They came to the conclusion that different rater groups provide differing levels of reliable feedback, probably because the number of “agendas” lurking within the various types of raters. The least reliable are the subordinates, followed by the peers, and then the managers, the most reliable rater group.

One way to address rater unreliability is to increase the size of the group (another might be rater training, for example). Usually there is only one manager and best practice is to invite all direct reports (who meet the tenure guidelines), so the main question is the number of peers. This research suggests that 7-9 is where we need to aim, noting also that that is the number of returns needed, so inviting more is probably a good idea if you expect less than a 100% response rate.

Another potential rater group is external customers. Recently I was invited to participate in a forum convened by the American Board of Internal Medicine (ABIM) to discuss the use of multisource feedback in physician recertification processes. ABIM is one of 24 member Boards of the American Board of Medical Specialties (ABMS), which has directed that some sort of multisource (or 360) feedback be integrated into recertification.

The participants in this forum included many knowledgable, interesting researchers on the use of 360 in the context of medicine (a whole new world for me, which was very energizing). I was invited to represent the industry (“outside) perspective. One of the presenters spoke to the challenge of collecting input from their customers (i.e., patients), a requirement for them. She offered up the number of 25 as the number of patients needed to create a reliable result, using very similar rationale as Greguras and Robie regarding the many individual agendas of raters.

Back to LinkedIn, there was then this opinion:

I agree that having too many raters in any one rater group does dilute the feedback and make it much harder to see subtleties. There is also a risk that too many raters may ‘drown out’ key feedback.

This is when my head started spinning like Linda Blair in The Exorcist.  This perspective is SO contrary to my 25 years of experience in this field that I had to prevent myself from discounting it as my head continued to rotate.  I have often said that a good day for me includes times when I have said, “Gee, I have never thought of (insert topic) in that way.” I really do like hearing new and different views, but it’s difficult when they challenge some foundational belief.

For me, maybe THE most central tenet of 360 Feedback is the reliance on rater anonymity in the expectation (or hope) that it will promote honesty. This goes back to the first book on 360 Feedback by Edwards and Ewen (1996) where 360’s were designed with this need for anonymity being in the forefront. That is why we use the artificial form of communication of using anonymous questionnaires and usually don’t report in groups of less than 3. We know that violations of the anonymity promise result in less honesty and reduced response rates, with the grapevine (and/or social media) spreading violated trust throughout the organization.

The notion that too many raters will “drown out key feedback” seems to me to be a total reversal of this philosophy of protecting anonymity. It also seems to place an incredible amount of emphasis on the report itself where the numbers become the sole source of insight. Other blog entries of mine have proposed that the report is just the conversation starter, and that true insight is achieved in the post-survey discussions with raters and manager.

I recall that in past articles (see Bracken, Timmreck, Fleenor and Summers, 2001) we made the point that every decision requires what should be a conscious value judgment as to who the most important “customer” is for that decision, whether it be the rater, ratee, or the organization. For example, limiting the number of raters to a small number (e.g., 5 per group or not all Direct Reports) indicates that the raters and organization are more important than the ratee, that is, that we believe it is more important to minimize the time required of raters than it is to provide reliable feedback for the ratee. In most cases, my values cause me to lobby on behalf of the ratee as the most important customer in design decisions.  The time that I will rally to the defense of the rater as the most important customer in a decision is when anonymity (again, real or perceived) is threatened. And I see these arguments for creating more “insight” by keeping rater groups small or subdivided are misguided IF these practitioners share the common belief that anonymity is critical.

Finally (yes, it’s time to wrap this up), Larry Cipolla, an extremely experienced and respected practitioner in this field, offers some sage advice with some comments, including the folly of increasing rater group size by combining rater groups. As he says, that is pure folly. But I do take issue with one of his practices:

We recommend including all 10 raters (or whatever the n-count is) and have the participant create two groups–Direct Reports A and Direct Reports B.

This seems to me to be a variation on the theme of breaking out groups and reducing group size with the risk of creating suspicions and problems with perceived (or real) anonymity. Larry, you need to show that doing this kind of subdividing creates higher reliability in a statistical sense that can overcome the threats to reliability created by using smaller N’s.

Someone please stop my head from spinning. Do I just need to get over this fixation with anonymity in 360 processes?

References

Bracken, D.W., Timmreck, C.W., and Church, A.H. (2001). The Handbook of Multisource Feedback. San Francisco: Jossey-Bass.

Bracken, D.W., Timmreck, C.W., Fleenor, J.W., and Summers, L. (2001). 360 feedback from another angle. Human Resource Management, 1, 3-20.

Edwards, M. R., and Ewen, A.J.  (1996). 360° Feedback: The powerful new model for employee assessment and performance improvement. New York: AMACOM.

Farr, J.L., and Newman, D.A. (2001). Rater selection: Sources of feedback. In Bracken, D.W., Timmreck, C.W., and Church, A.H. (eds.), The Handbook of Multisource Feedback. San Francisco: Jossey-Bass.

Greguras, G.J., and Robie, C. (1998).  A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.

©2012 David W. Bracken

As an alumnus of Dartmouth College, I receive the Alumni Magazine whose current issue contains an interview with the new(ish) president, Jim Kim (see http://dartmouthalumnimagazine.com/the-dam-interview/  if you can’t control yourself).  A couple things in the interview caught my attention, including this statement:

The folks in leadership studies at Tuck have said the one thing that is critical for the development of better leaders is self-awareness, the so-called 360-degree analysis. The challenge for us is to structure the kind of education that will lead to the graduation of young people with a clearer sense of what it will take for them to be effective human beings.

Of course, the “360” part is interesting in itself, though I’m not sure what the “so-called” part is all about.

Is self-awareness the most critical of all leadership qualities? My model of leadership behavior change includes awareness, followed by acceptance, as the “keystones” to creating sustainable change.  Organizations are also in constant flux and in need of change, and organizations need some way to create awareness.  Dashboards are a way that organizations become of area in which they are succeeding and failing, and therefore drive change. For the individual leader, the 360 feedback process may be the most powerful dashboard if done correctly, at least on the “how” side of the performance equation (versus the “what”).

Another argument for the importance of awareness came to my attention during the current Republican primary contest. One pundit, in comparing the field of contenders, offered an observation that, except for Rick Santorum, the other players seem to be lacking this sense of self that Dr. Kim alludes to in his quote. One symptom of that lack of self is the constant and repeated use of “Reagan Republican” by almost all the candidates to describe themselves. I even saw a parody of a contest of the candidates as to who could say the name “Reagan” the most times in 10 seconds.

While I’m at it, there was one other quote from the interview that is worth sharing:

It’s fairly well known now that I have a leadership coach, Marshall Goldsmith, who was recently ranked one of the world’s top-10 thought leaders and who also teaches at Tuck. He took me on as a pro bono case. In Marshall’s book, What Got You Here Won’t Get You There, he lists the 20 most common mistakes that CEOs make. Probably the biggest mistake is adding too much value. I didn’t understand that in the beginning, but I sure do now.

You may know that I am a follower of Marshall’s and the book is one I have reviewed and passed along to others (including my family members).  Dr. Kim offers up another important leadership characteristic or, in this case, flaw that plague leaders as they move up the organization. I have compared Marshall’s list of 20 pitfalls to be similar in spirit to the derailers described many years ago by Morgan McCall and associates at the Center for Creative Leadership, though the specific content is different. But both can also be useful content for 360 feedback processes.

Is it time to go and defog your mirror and test your self awareness?   Oh, and remember that what is in your mirror may be closer than it appears.

©2012 David W. Bracken

The world is a dangerous place to live; not because of the people who are evil, but because of the people who don’t do anything about it.

-Albert Einstein
I hadn’t heard this quote before this weekend. It happened to be on a sign carried by a lone protester outside the entrance to the Penn State football game, standing by the Joe Paterno statue. Needless to say, his presence and message wasn’t appreciated by some of the PSU faithful, though he stated that he was once “one of them” but now had a different perspective as a family man in the wake of these recent events.

Another article in today’s (Nov 12) NY Times also caught my eye in what I felt had a related message, titled “For Disabled Care Complaints, Vow of Anonymity Was False.” (http://www.nytimes.com/2011/11/12/nyregion/ombudsmen-gave-whistle-blowers-names-to-state-agency.html?src=me&ref=nyregion).  A spokesman for the agency, Travis Proulx, said in an interview in August that “there is no confidentiality for any employee who is reporting abuse or neglect, even to the ombudsman.” Is it any wonder that people are afraid to step forward?

Organizations, including universities, are in many ways closed systems with their own methods for defining and living values (http://www.nytimes.com/2011/11/12/us/on-college-campuses-athletes-often-get-off-easy.html?ref=ncaafootball).   See also the recent news story about the Texas judge who has been exposed via YouTube of his own brand of values inside his “organization,”, i.e., his family (http://www.nytimes.com/2011/11/13/us/ruling-against-judge-seen-beating-daughter.html?scp=1&sq=texas%20judge%20belt&st=cse).  Without getting into legalities and regulation and the such, let us focus on the fact that organizations (of any kind) need some sort of internal processes, formal and/or informal, to define proper behavior and to rectify instances of wrongdoing.

Whatever the unit of analysis, the definition of “evil” is a very subjective process. In an earlier blog, I pointed to some research that suggested that some questionable practices are more acceptable in some industries than in others (http://dwbracken.wordpress.com/2011/03/15/what-is-normal/).  And I do believe that organizations have the right to define their values and to hold employees accountable for behaving in ways consistent with those values. Some actions are egregious that they are universally rejected (at least within certain cultures), including those exhibited by psychopaths as described in the book, Snakes in Suits.

One of the many benefits of doing a system-wide (e.g., company, department) 360 feedback process is the opportunity it creates for miscreants to be identified through anonymous input from coworkers. If system-wide, they hopefully also detect psychopaths and the such who are also very skilled at escaping detection. Unlike other “whistle blowing,” 360’s rely on a consensus from feedback providers that theoretically protects both the raters and the ratees. The data generated by 360’s is reported in aggregate form, usually requiring a minimum of three respondents to create a mean score. Assuming the organization has access to these scores, they can be analyzed to detect particularly low mean scores that indicate that the leader in question is being cited by multiple coworkers as being out of synch with the rest of the organization.

So what do we need to do to make our 360 processes useful for detecting misbehavior and protecting both the raters and the ratees alike? Some suggestions include:

• Be clear as to the purpose of the process
• Require participation by all organizational leaders
• Give access to results to the organization (including HR and management)
• Strictly adhere to minimum group size requirements for reporting results (e.g., minimum of 3)
• Use a well designed behavioral model to form the basis for the content.
• Include write in comments.
• Train users (managers, HR) on the proper interpretation and use of results
• Administer on a regular (annual) basis
• Immediately address instances of leaders seeking retribution against raters (real or inferred)

Any other suggestions?

©2011 David W. Bracken

The Sunday New York Times Business section typically has a feature called “Corner Office” where a CEO is interviewed. These CEO’s seem to often be from small businesses. The one today (October 16, 2011), for example, is the CEO of a 25-person, $6 million consulting firm. The questions are often the same, having to do with the process of becoming a leader, lessons learned, hiring and promotion strategies. I have referenced these in a couple earlier blogs since they touch on topics relative to behavior change, leadership development and culture change that are relevant to aspects of 360 processes.

In today’s column, the CEO was asked about how he hires. He notes that part of the interview often includes asking the applicant to rate him/herself on a five point scale regarding various areas of knowledge and experience from their resume. If the applicant rates themselves as a “5,” he then asks if there is nothing else that they could learn about whatever it is. Of course, they say, “Oh, no, no. There is.” To which the CEO asks, “then why did you rate yourself a five?”  And he goes on to say he has never hired someone who rates themself as a five.

While this CEO’s decision not to hire these high self raters may seem arbitrary, think of the culture he is trying to create by the people he selects and the message this communicates to new hires during the process. He says that he appreciates humbleness and their understanding that they don’t know everything.  (In an earlier column, the CEO asks applicants if they are “nice,” and then explains what “nice” means in their culture, i.e., a good team player.)

(Someone told me of a senior executive who greeted applicants in the lobby and asked them whether they should take the stairs or elevator. If they said elevator, he didn’t hire them. That seems less job related, but is our “5″ CEO doing a similar thing? Food for thought.)

We don’t use 360’s to hire people (though I have heard of multisource reference checks  from  past coworkers that are being positioned as 360’s), but we do have an opportunity with 360’s to create or support a culture when they involve many people. But we also know that 360’s are notorious for having severe leniency, i.e., mostly 4’s and 5’s on a five point scale.

Do all these 5’s that we collect mean that our leaders can’t get any better at what they do? Of course not. But that seems to be the message that we allow and even reward (even if not tangibly).

The vast majority of 360 processes use an Agree/Disagree (Likert) scale where “Strongly Agree” is scored as a “5” (scales that score it as a “1” seem confusing and counterintuitive to me).  The VAST majority of processes also do not include rater training that could be used to help raters (and ratees for that matter) to stop attaching any meaning to “Strongly Agree” that they wish. Which they currently do.

I have used a rating scale where “5” is defined as “role model, in the top 5-10%” that attempts to create a frame of reference for raters (and ratees) that does help reduce leniency effects.

What if we defined “5” as “can’t get any better” or something equivalent to that. I think “role model” implies that this person can be a teacher as well as example to others, and perhaps doesn’t need to get better (i.e., focus on other areas of improvement).  Some raters will undoubtedly ignore those directions, but rater training can help drill in the need for everyone to reconfigure their conceptualization of what optimal behavior is and, by the way, foster the learning and development culture that our CEO seems to be nurturing.

A recalibration of rating scales is badly needed in this field. We need to stop raters from giving all “5’s” and from ratees giving self ratings of all “5’s”.  With our current mentality on rating scales, there is really nothing to stop rating inflation. It should be no surprise that senior leaders find it to be difficult to use and support our 360 programs.

©2011 David W. Bracken

I don’t know why I feel compelled to respond to what I see are unreasonable positions (primarily in LinkedIn discussions). But I do, and this blog gives me a vehicle for doing so without taking up a disproportionate amount of air time on that forum.

So what got me going this time? A LinkedIn discussion (that I started on the topic of 360 validity) got diverted into the topic of “proper” use of 360 feedback (development vs decision making).  The particular comment that got me going was, “I believe these assessments should be used for development – full stop.”  (Virtually 100% of 360 processes are used for development, but the context indicates that he meant “development only.”) Having lived and worked in London for a while, I realized (or realised) that the “full stop” has the same meaning as “period,” implying end of sentence and, with emphasis, no more is worth saying.  By the way, I am using this person only as an example of the many, many individuals who have expressed similar dogmatic views on this topic.

There are probably a few things that are appropriate to put a “full stop” on. That would be an interesting blog for someone, e.g., would we include the Ten Commandments? “Thou Shall Not Kill. Full stop.”  Hmmm… but then we have Christians who believe in capital punishment, so maybe it’s only a partial stop (or pause)?  Like I said, I will let someone else take that on.

Are the physical sciences a place for “full stops?”   Like, “The world is flat. Full stop.”  “ The Sun revolves around the Earth. Full stop.” Just this last week, we were presented with the possibility that another supposedly immutable law is under attack, i.e., “Nothing can go faster than the speed of light. Full stop.”  Now we have European scientists who have observed neutrinos apparently traveling faster than the speed of light and are searching for ways to explain and confirm it. If found to be true, it would challenge many of the basics of physics, opening the door to time travel, for example.  The fact that some scientists are apparently challenging the “full stop” nature of the Theory of Relativity is also fascinating, if only for the reason that they are open to exploring the supposedly impossible. And, by the way, are begging for others to challenge and/or replicate their findings.

I firmly believe that the social sciences have no place for “full stops.”  To me, “full stop” means ceasing to explore and learn. It seems to indicate a lack of openness to considering new information or different perspectives.

I suspect there are many practitioners in the “hard” sciences who question whether what we do is a “science” at all. (I think I am running out of my quota of quotations marks.)  Perhaps they see our work with understanding human behavior as a quest with no hope of ever having answers. That’s what I like about psychology. We will never fully know how to explain human behavior, and that’s a good thing. If we can explain it, then we probably could control it. I think that is a scary thought. BUT we do try to improve our understanding and increase the probabilities of predicting what people will do. That is one of the basic goals of industrial/organizational psychology.

(I have been known to contend that what we do is harder than rocket science because there are no answers to what we do, only probabilities.  The truth is that even the hard sciences have fewer “full stops” than even they would like. I just finished reading a book about the Apollo space program, Rocket Men, and it is very interesting to know how many “hard stops” that used to exist were bashed (e.g., humans can’t live in weightlessness, the moon’s crust will collapse if we try to land on it. Insert “hard stops” appropriately), how much uncertainty there was, and how amazing the accomplishment really was.  I also learned that one of the reasons the astronauts’ visors were mirrored was so that aliens couldn’t see their faces. Seriously.)

Increasing probabilities for predicting and influencing employee behavior requires that we also explore options.  I can’t see how it is productive to assert that we know the answer to anything, and that we shouldn’t consider options that help us serve our clients, i.e., organizations, more effectively.

On top of all that, the most recent 3D Group benchmark study indicates that about one third of organizations DO use 360 data for some sort of administrative purpose, and that almost certainly understates the real numbers. What do we tell those organizations? That they should cease doing so since our collective wisdom says that there is no way they can actually be succeeding? That we cannot (or should not) learn from what they are doing to help their organizations make better decisions about their leaders? That a few opinions should outweigh these experiences?

I don’t get it. No stop.

©2011 David W. Bracken

My friend, Joan Glaman, dropped me a note after my last blog, (http://dwbracken.wordpress.com/2011/08/30/thats-why-we-have-amendments/ ) with this suggestion:

“I think your closing question below would be a great next topic for general discussion: ‘Under what conditions and for whom is multisource feedback likely to be beneficial?’”

To refresh (or create) your memory, that question that Joan cites is from the Smither, London and Reilly (2005) meta analysis. The article abstract states:

“…improvement is most likely to occur when feedback indicates that change is necessary, recipients have a positive feedback orientation, perceive a need to change their behavior, react positively to the feedback, believe change is feasible, set appropriate goals to regulate their behavior, and take actions that lead to skill and performance improvement.”

Before we answer Joan’s question, we should have a firm grasp on what we mean by “beneficial.” I don’t think we all would agree on that in this context.  Clearly, Smither et al. define it as “improvement,” i.e., positive behavior change. That is the criterion (outcome) measure that they use in their aggregation of 360 studies. I am in total agreement that behavior change is the primary use for 360 feedback, and we (Bracken, Timmreck, Fleenor and Summers, 2001) defined a valid 360 process as one that creates sustainable behavior change in behaviors valued by the organization.

Not everyone will agree that behavior change is the primary goal of a 360 process. Some practitioners seem to believe that creating awareness alone is a sufficient outcome since they do not support any activity or accountability, proposing that simply giving the report to the leader is going far enough and in fact discourage the sharing of results with anyone else.

If you will permit a digression, I will bring to your attention a recent blog by Sandra Mashihi (http://results.envisialearning.com/5-criteria-a-360-degree-feedback-must-meet-to-be-valid-and-reliable/) where one of her lists of “musts” (arrrgh!) is criterion related validity, which she defines as, …does the customized instrument actually predict anything meaningful like performance?” Evidently she would define “beneficial” then to not be behavior change but to be able to measure performance to make decisions about people.  This testing mentality just doesn’t work for me since 360’s are not tests (http://dwbracken.wordpress.com/2010/08/31/this-is-not-a-test/) and it is not realistic to expect them to predict behavior, especially if we hope to actually change behavior.

Let’s get back to Joan’s question (finally). I want to make a couple comments and then hopefully others will weigh in. The list of characteristics that Smither et al provide in the abstract is indeed an accumulation of individual and organizational factors. This is not an “and” list that says that a “beneficial” process will have all these things. It an “or” list where each characteristic can have benefits.  The last two, (set goals and take actions) can be built into the process as requirements regardless of whether the individual reacts positively and/or perceives the need to change. Research shows that follow up and taking action are powerful predictors of behavior change, and I don’t believe that it is important (or matters) to know if the leader wants to change or not. What if he/she doesn’t want to change? Do they get a pass? Some practitioners would probably say, yes, and point to this study as an indication that it is not worth the effort to try to get them to change.

I suggest that this list of factors that lead to behavior change are not independent of each other. In our profession, we speak of “covariates”, i.e., things that are likely to occur together across a population. A simple example is gender and weight, where men are, on average, heavier than women. But we don’t conclude that men as a gender manage their weight less well than women, it’s due to being taller (and other factors, like bone structure).

My daughter, Anne, mentioned in passing an article she read about people who don’t brush their teeth twice a day having a shorter life expectancy than those who do.  So the obvious conclusion is that brushing teeth more often will make us live longer.  There is certainly some benefit to regularly brushing teeth, but it’s more likely that there are covariates of behavior for people that don’t have good dental hygiene that have a more direct impact on health.  While I don’t have data to support it, it seems likely that people who don’t brush regularly also don’t go to the dentist regularly for starters.  It seems reasonable to surmise that, on average, those same people don’t go to their doctor for a regular checkup.

My hypothesis is that 360 participants who aren’t open to feedback, don’t perceive a need to change, don’t feel that they can change, etc., are also the people who are less likely to set goals and take action (follow up) if given the option to not do those things.  In other words, it’s not necessarily their attitudes that “cause” lack of behavior change, but the lower likelihood that they will do what is necessary, i.e., set goals and follow through, in order to be perceived as changing their behavior. Those “behaviors” can be modified/changed while their attitudes are likely to be less modifiable, at least until they have had a positive experience with change and its benefits.

One last point of view about “beneficial.” Another definition could be change that helps the entire organization. That is the focus of the recent publication by Dale Rose and myself, where (in answer to Joan’s question) we state:

“…four characteristics of a 360 process that are required to successfully create organization

change, (1) relevant content, (2) credible data, (3) accountability, and (4) census participation…”

We go on to offer the existing research that supports that position, and the wish list for future research. One way of looking at this view of what is “beneficial” is to extrapolate what works for the individual and apply it across the organization (which is where the census (i.e., whole population) part comes into play.)

I will stop there, and then also post this on LinkedIn to see if we can get some other perspectives.

Thanks, Joan!

©2011 David W. Bracken

I used my last blog (http://dwbracken.wordpress.com/2011/08/09/so-now-what/)  to start LinkedIn discussions in the 360 Feedback and I/O Practitioners group, asking the question: What is a “valid” 360 process?  The response from the 360 group was tepid, maybe because the group has a more general population that might not be that concerned with “classic” validity issues (which is basically why I wrote the blog in the first place).  But the I/O community went nuts (45 entries so far) with comments running the gamut from constructive to dismissive to deconstructive.

Here is a sample of some of the “deconstructive” comments:

…I quickly came to conclusion it was a waste of good money…and only useful for people who could (or wanted to) get a little better.

It is all probably a waste of time and money. Good luck!!

There is nothing “valid” about so-called 360 degree feedback. Technically speaking, it isn’t even feedback. It is a thinly veiled means of exerting pressure on the individual who is the focal point.

My position regarding performance appraisal is the same as it has been for many years: Scrap It. Ditto for 360.

Actually, I generally agree with these statements in that many 360 processes are a waste of time and money. It’s not surprising that these sentiments are out there and probably quite prevalent. I wonder, though, if we are all on the same page. In another earlier blog, I suggested that discussions about the use and effectiveness of 360’s should be separated by those that are designed for feedback to a single individual (N=1) and those that are designed to be applied to groups (N>1).

But the fact is that HR professionals have to help their management make decisions about people, starting with hiring and then progressing through placement, staffing, promotions, compensation, rewards/recognition, succession planning, potential designation, development opportunities, and maybe even termination.

Nothing is perfect, especially so when it comes to matters that involve people. As an example, look to the U.S. Constitution, an endearing document that has withstood the test of time. Yet the Founding Fathers were the first to realize that they needed to make provisions for the addition of amendments to further make refinements. Of course, some of those amendments were imperfect themselves and were later rescinded.

But we haven’t thrown out the Constitution because it is imperfect.  Nor do we find it easy to come to agreements what the revisions should be.  But one of the many good things about humans is a seemingly natural desire to make things better.

Ever since I read Mark Edwards and Ann Ewen’s seminal book, 360 Degree Feedback, I have believed that 360 Feedback has the potential to improve personnel decision making when done well. The Appendix of The Handbook of Multisource Feedback is titled, “Guidelines for multisource feedback when used for decision making,” coauthored with Carol Timmreck, where we made a stab at defining what “done well” can mean.

In our profession, we have an obligation to constantly seek ways of improving personnel decision making. There are two major needs we are trying to meet, which sometimes cause tensions. One is to provide the organization with more accurate information on which to base these decisions, which we define as increased reliability (accurate measurement) and validity (relevant to job performance). Accurate decision making is good for both the organization and the individual.

The second need is to simultaneously use methods that promote fairness. This notion of fairness is particularly salient in the U.S. where we have “protected classes” (i.e., women, minorities, older workers), but hopefully fairness is a universal concept that applies in many cultures.

Beginning with the Edwards & Ewen book and progressing from there, we can find more and more evidence that 360 done well can provide decision makers with better information (i.e., valid and fair) than traditional sources (e.g., supervisory evaluations).  I actually heard a lawyer state that organizations could be legally exposed for not using 360 feedback because is more valid and fair than methods currently in use.

I have quoted Smither, London and Reilly (2005) before, but here it is again:

We therefore think it is time for researchers and practitioners to ask “Under what conditions and for whom

is multisource feedback likely to be beneficial?” (rather than asking “Does multisource feedback work?”).

©2011 David W. Bracken

This is the one year anniversary of this blog. This is the 44^th post.  We have had 2,026 views, though the biggest day was the first with 38 views.  I have had fewer comments than I had hoped (only 30), though some LinkedIn discussion have resulted. Here is my question: Where to go from here? Are there topics that are of interest to readers?

Meanwhile, here is my pet peeve(s) of the week/month/year:  I was recently having an exchange with colleagues regarding a 360 topic on my personal Gmail account and up pops ads on the margin for various 360 vendors (which is interesting in itself), the first of which is from Qualtrics (www.qualtrics.com) with the heading, “Create 360s in Minutes.”

The topic of technology run amok has been covered before here (When Computers Go Too Far, http://wp.me/p10Xjf-3G), my peevery was piqued (piqued peevery?) when I explored their website and saw this claim:  USE VALIDATED QUESTIONS, FORMS and REPORTS.”

What the heck does that mean?  What are “validated” forms and reports, for starters?

The bigger question is, what is “validity” in a 360 process?  Colleagues and I (Bracken, Timmreck, Fleenor and Summers, 2001; contact me if you want a copy) have offered up a definition of validity for 360’s that holds that it consists of creating sustainable change in behaviors valued by the organization.  Reliable items, user friendly forms and sensible reports certainly help to achieve that goal, but certainly cannot be said to be “valid” as standalone steps in the process.

The Qualtrics people don’t share much about who they are. Evidently their founder is named Scott and teaches MBA’s.  They appear to have a successful enterprise, so kudos!  I would like to know how technology vendors claim to have “valid” tools and what definition of validity they are using.

Hey maybe I will get my 31^st comment?

©2011 David W. Bracken

My colleague, Jeff Saltzman, has a great blog that is much more diverse than mine (http://jeffreysaltzman.wordpress.com ). His most recent entry begins with this little gem of a story that I want to plagiarize and take in my own direction:

”Which is more important, the sun or the moon?” A citizen of a small town not noted for its intellectual prowess asked. “Why the moon of course,” was the reply. “It shines at night when it is needed. The sun shines only during the day, when there is no need of it at all!” (Ausbel, N., A Treasury of Jewish Folklore, 1948)

I have touched on the topic of importance in some past blogs (http://dwbracken.wordpress.com/2011/04/07/worst-is-not-first/), and the folly of asking raters what is “important”. This little story made me think of that issue once again from a slightly different angle. My stance has been, and still is, that raters are in a very poor position to judge the importance of a competency/behavior in the context of the need of the ratee and the organization.

There is really no way to know what is going through a rater’s mind if/when we ask him/her to give use importance ratings. There may be some research on this question (e.g., correlation between importance and effectiveness ratings), but I will hazard a guess that importance ratings are more a function of rater needs than the needs of the ratee or organization/team.

Jeff’s story also makes me wonder how qualified raters are to provide importance ratings when they are most likely not given any instruction as to what “importance” means (as rater training might attempt to do). And their rationale for importance ratings may well be as convoluted as the small town citizen’s is.

The question of importance is useful in helping prioritize actions. So, if it is not the raters who should indicate importance, who is it? The manager (“boss”), of course, partnering with the ratee. Hopefully the boss and ratee have a history of development discussions on a personal level, and about organization/team priorities to create alignment. If they have not been having those discussions, maybe a 360 process tied to performance management and development might create some mutual accountability for doing so.

The importance of the “boss” in the 360 process and employee development in general is so critical that it boggles the mind to think of 360’s that totally bypass (exclude) the manager. I will equally dismayed to read of a major 360 process describe on LinkedIn that makes boss input optional.  Really? I have always thought that manager input is the most useful feedback many ratees get out of 360’s, to the extent that a best practice is to require that the boss complete their input in order for a report to be generated.

I will go as far as to say the manager ratings are more important than participant self-ratings. Ideally both will happen but, as I mentioned in a recent blog, self- ratings are more an indication of commitment to the process than a true evaluation of self competence in many, many cases.  I will acknowledge that sometimes bosses use their ratings to send a message to the ratee, but even then the resulting discussion is often very enlightening for the ratee.

©2011 David W. Bracken

This is a “two sided” blog entry, like those old 45 rpm records that had hit songs on both sides (think  “We Can Work It Out”/”Daytripper” by the Beatles),though my popularity may not be quite at their level.  This is precipitated by a recent blog (and LinkedIn discussion entry) coming from the Envisia people. The blog entry is called, “Does 360-degree feedback even work?” by Sandra Mashihi and can be found at http://results.envisialearning.com/.   It would be helpful if you read it first, but not necessary.

Sandra begins by citing some useful research regarding the effectiveness of 360 processes. And she concludes that sometimes 360’s “work” and sometimes not.  Her quote is, “Obviously, the research demonstrates varied results in terms of its effectiveness.”

What is frustrating for some of us are the blanket statement about failures (and using terms like “obvious”) without acknowledging that many 360’s are “built to fail.” This is the main thesis of the article Dale Rose and I just published in the Journal of Business and Psychology. http://www.springerlink.com/content/85tp6nt57ru7x522/

http://www.ioatwork.com/ioatwork/2011/06/practical-advice-for-designing-a-360-degree-feedback-process-.html

Dale and I propose four features needed in a 360 process if it is likely to create sustainable behavior change:

1)      Reliable measurement: Professionally developed, custom designed instruments

2)      Credible data: Collecting input from trained, motivated raters with knowledge of ratees

3)      Accountability: Methods to motivate raters and ratees to fulfill their obligations

4)      Census participation: Requiring all leaders in an organizational unit to get feedback

We go on to cite research that demonstrates how the failure to build these features into 360 can, in some cases, almost guarantee failure and/or the ability to detect behavior change when it does occur. One such feature, for example, is whether the ratee follows up with raters (which I have mentioned in multiple prior blogs). If/when a 360 (or a collection of 360’s, such as in a meta analysis) is deemed a “failure”, I always want to know things such as whether raters were trained and whether follow up was required, for starters.

We are leaning more and more about the facets that increase the probability that behavior change will occur as a result of 360 feedback. Yet all too often these features are not built into many processes, and practitioners are surprised (“shocked, I’m shocked”) when it doesn’t produce desired results.

Sandra then goes on to state: “I have found 360-degree feedback worked best when the person being rated was open to the process, when the company communicated its purpose clearly, and used it for development purposes.” I assume that she means “development only” since all 360’s are developmental.  I definitely disagree with that feature. 360’s for “development (only) purposes” usually violate one or more of the 4 features Dale and I propose, particularly the accountability one. They often do not generate credible data because too few raters are used, even the best practice of including all direct reports.

The part about “being open to the process” is where I get the flip side of my record, i.e., don’t hurt my feelings.  In one (and only one) way, this makes sense. If the ratee doesn’t want to be in a development-only process, then by all means don’t force them. It is a waste of time and money. On the other hand, all development only processes are a waste of money in my opinion for most people. (And, by the way, development only is very rare if that means that no decisions are being made as a result.)

But if we ARE expecting to get some ROI (such as sustained behavior change) from our 360’s, then letting some people to opt out so their feelings aren’t hurt is totally contrary to helping the organization manage its leadership cadre. Intuitively, we should expect that those who opt out are the leaders that need it the most, who know that they are not effective and/or are afraid to be “discovered” as the bullies, jerks, and downright psychopaths that we know exist out there.

I have some fear that this fear of telling leaders that they are less than perfect is stemming from this troubling trend in our culture where everyone  has to succeed. I think that the whole “strengths” movement is a sign of that.

Over the last couple of weeks, I have seen a few things that further sensitized me to this phenomenon. One big one is this article in The Atlantic: http://www.theatlantic.com/magazine/archive/2011/07/how-to-land-your-kid-in-therapy/8555/1/.  Protecting our children from failure is not working. Protecting our leaders from failure is also dooming your organization.

I swear I never watch America’s Funniest Videos, but during a rain delay of a baseball game recently, I did stumble upon it and succumbed. AFV is all about failure, and I’m not so sure that people always learn from these failures. But one video I enjoyed showing a 2 year old boy trying to pour apple juice from a BIG bottle into a cup. He put the cup on the floor and totally missed the first two times (with the corresponding huge mess). As a parent and grandparent, I was quite amazed that the person behind the camera just let it happen. But on the third try, the task was accomplished successfully, followed by applause and smiles! There was a huge amount of learning that occurred in just a minute or two because the adults allowed it to happen, with a bit of a mess to clean up.

How many of us would have just poured the juice for him? His learning isn’t over; he will make more mistakes and miss the cup occasionally. But don’t we all.

As a parting note, Dale and I support census participation for a number of reasons, one of which is the point I have already made about otherwise missing the leaders that need it most. We also see 360’s as a powerful tool for organizational change, and changing some leaders and not others does not support that objective. Having all leaders participate is tangible evidence that the process has organization support and is valued. Finally, it creates a level playing field for all leaders for both evaluation and development, communicating to ALL employees what the organization expects from its leaders.

©2011 David W. Bracken