Strategic 360s

Making feedback matter

Archive for January 2011

Who is the customer? Take Two

leave a comment »

[tweetmeme source=”anotherangle360″]

In an earlier blog, I asked the question, “Who is the customer in a 360 process?”  The particular focus of that discussion was the length of 360 questionnaires.

I have recently been exploring the websites of various 360 Feedback providers to see the products/services that are offered, and how they are positioned. I was surprised by how many vendors are offering processing services without consulting services, which I think relates back to the potential problems that computer-driven solutions can cause that were covered a couple blogs ago.

What I would really like to briefly address, stemming from this search exercise, is the report formats and associated decisions in their design. One of the many decisions in designing a report is whether to show the actual frequency of responses. In my search of 360 websites, when report samples were provided they more often than not showed the mean score for each rater group (e.g., self, boss, direct reports, peers, customers) but not how the average was derived (i.e., the rating distribution).

From experience, I know where this decision comes from. Typically, the rationale is that showing the frequencies will potentially draw attention to outliers (a single rating, usually at the low end of the scale) and cause problems if the ratee misuses that information. Misuse can come in the form of making assumptions as to who gave the rating, and/or exact some form of retribution on the supposed outlier.

These things do happen. My question is whether the best solution to this problem is to deny this potentially useful data to the ratee and other consumers of the report, such as their manager and possibly a coach.

When discussing the question of the length of the survey, I proposed that short (less than 25 items, for example) surveys can be a sign that the rater is a more important “customer” of a 360 process than the ratee. The abbreviated survey makes the task supposedly easier for the raters but denies the ratee from having useful data on a broader range of behaviors that may be useful depending on his/her situation.

Similarly, not showing frequencies again identifies the rater as being more important than the ratee. Not providing the frequencies somehow protects the rater from potential abuse. On the other hand, providing that information to the ratees can be extremely useful in understanding the degree of consensus among raters. (Some reports provide some index of rater agreement in lieu of rating distributions, but I have consistently found those to be almost useless and frequently misunderstood).

Distributions can also help the ratee and other uses to see how outliers do have a major impact on mean scores, especially when the N is small (which it often is with 360’s). I have also found it useful to be able to see cases where there is one outlier on almost every question. When possible, I have asked the data processor to verify that the outlier was the same person (without identifying who it was), and I informed the ratee and his/her manager that scores have been affected by one person who has abused their role as a feedback provider. I also have provided counsel on how to use that information, including not making assumptions as to who the outlier is and to not attempt to discover who it was.

With one client that was provided with rating distributions, the report had this kind of pattern with one outlier on every item. (I have told this story before but in a different context.) Since he met with his raters (a best practice) and shared his report (another best practice), he felt compelled to mention that apparently someone had some problems with his management style and that he would appreciate it if the person, whoever it was, would come talk with him sometime. He left it at that.  Sure enough, a member of the team did come to see him and, with embarrassment, confessed that he had accidentally filled it out backwards (misread the scale).  Think how this helped the manager in so many ways to not have to make assumptions as to the cause of the feedback results. If he did not have the detail of the frequencies he (and his boss/coach) would not know how the averages had been artificially affected. It is also one more reason why 360’s should be used with some judgment, not just treated as a “score.”

We do need to protect raters from being abused. We also need to help them feel safe from identification so that they will continue to be honest in their responses. One way to approach that is to ensure that managers and coaches reinforce to the ratees the proper ways to read, interpret and use the feedback. There should also be processes in place to identify ratees who do behave improperly and to hold them accountable for their actions.

Ratees should be the most important “customer” in 360 processes. They need to understand how their raters responded and, by the way, on a sufficient number of items to apply to their situation. Design and implementation decisions that treat raters as being more important than ratees are misguided.

©2011 David W. Bracken

When Computers Go Too Far

leave a comment »

[tweetmeme source=”anotherangle360″]

In my last blog entry, I highlighted a recent article by Dale Rose and colleagues that largely focuses on the use of technology in supporting 360 Feedback processes; I hope you had a chance to read it. Here is the link again: http://www.siop.org/tip/jan11/04rose.aspx

One of the points the article made was how technology can be used to basically bypass some important  steps in a 360 process. One that blows my mind is offering raters a list of prefab write-in comments. Give me a break.

Rose et al speak to data overkill that can result in overly complex and lengthy reports.  The inverse of that is a capability they don’t mention in the article, namely to create an extremely short report (a page or two)  that is supposed to “tell” the participant what the bottom line is, i.e., what the report says is most important to work on. In other words, users want to bypass reading the report at all, and just “give me the answer.”  Now, we do need to help people use the report to help distill the few messages that are relevant for their situation that will ideally lead to some development plans. But, in the end, no computer (or rater or coach, for that matter) can do that for them. (As I noted in an earlier blog, I believe it is misguided to ask raters or coaches to identify the most important development needs; that responsibility lies with the people who best know the ratee’s situation, which is usually the ratee and their manager.)

Another example Rose et al  did mention, and the one that motivated to write this blog entry, was how technology can be used to create development plans. I recall being in a sales meeting some time back at a consulting firm that will be left nameless. A senior manager said, “Hey wouldn’t it be great to create the capability for the computer to analyze the report and tell the leader what actions to take. People would love it!”  I think my response was something like, “Are you nuts?”  Maybe that’s one reason why I’m not there anymore.

From a sales perspective, in my defense we were trying to create a coaching practice to help leaders to do just that, i.e., properly read and use their feedback, partnering with the participant’s manager (boss) to create accountability and sustainable behavior change. Speaking of accountability and sustainability, how committed will a person be to a computer generated suggestion? It might as well be a horoscope or fortune cookie.

Speaking of computers, GPS devices have become very popular. They can be very helpful and I use mine often.  They also have some problems, one of which is that they are sometimes wrong. There are a couple of entertaining commercials on TV right now that use that theme to show the consequences of a GPS gone awry. Part of what is amusing about these commercials is how the drivers are totally reliant on the “data” coming from the computer and ignore reality.

If commercials are funny because drivers follow a GPS mindlessly, why isn’t it funny when we do the same with  computer-generated 360 reports that tell us what to do?  If anyone is laughing, it is the firms that are selling that stuff. (Let me be clear that I am not dissing development guides that list resources to choose from. I am criticizing being told, either by a computer or even a coach, which of those resources are the “silver bullet.”)

One capability that my GPS has is to display both the current (supposed) speed limit and my actual speed. Again, sometimes it is just plain wrong, maybe out of date or not able to know about construction zones, for example. In a 360 analogy, a speed limit is a kind of norm, that is, a comparison number. I am of the opinion that most (if not all) external 360 norms are likely to be irrelevant to the target organization and, therefore, each employee as well. In other words, they are often an inaccurate comparison. An internal norm, on the other hand, is a much more reliable comparison number that takes into consideration the uniqueness of the environment (terrain, weather, traffic?) and special situations (e.g., construction zones, speed traps).

My GPS actually goes beyond just displaying the two numbers (speed limit and actual speed) and makes a value judgment by flashing red when I exceed 3 mph faster than the limit. REALLY? Three miles per hour? And sometimes against an erroneous number to start with? There is a not-so-fine line between providing people with data to consider in regard to changing behavior versus telling them that what they have to do without considering the data accuracy and the context.

I have always maintained that 360’s should not be about the numbers. There ALWAYS needs to be a good dose of judgment used in interpreting and using the results. Yes, I am a proponent of using 360’s to help us make decisions. “Help” is the operative word; not to “make” decisions but to “help” us make better decisions along with other data, observations, history, values, and common sense.

No computer should do those things for us. And if/when we do ask computers to do the work for us, it is a sign of lack of commitment to maximizing the usefulness (read “validity”) of the 360 process.

©2011 David W. Bracken

Making Mistakes Faster

leave a comment »

[tweetmeme source=”anotherangle360″]

The primary purpose of this brief blog entry is to bring to your awareness a new article by Dale Rose, Andrew English, and Christine Thomas in The Industrial/Organizational Psychologist (TIP). I assume that a minority of readers of this blog receive TIP, or, if they do, have not had a chance to read this article. (The title would not immediately draw attention to the fact that the majority of the content is about 360 Feedback for starters.)

The article can be accessed at http://www.siop.org/tip/jan11/04rose.aspx.

As you will see, Dale and colleagues focus primarily on how technology has affected 360 Feedback processes, for good and bad. This should be required reading for practitioners in this field.

They reference a discussion Dale and I had on this blog about the “silly” rating format where raters can rate multiple ratees at the same time using a kind of spread sheet layout. They are correct that there is no research that we are aware of that studies the effects of rating formats like this on the quality of ratings and user reactions (including response rates, for example). We won’t rehash the debate here, but suffice to say that it is one area where Dale and I are in disagreement.

Other than that, I endorse his viewpoints about the pitfalls of technology. I recall when computers first became available to us to support our research. As we all struggled to use technology effectively, I remember saying that computers allow us to make mistakes even faster.

I will use my next blog to talk about, “When Computers Go Too Far,” which builds on some of Dale’s observations. Hope you will tune in!

©2011 David W. Bracken

There Are “Right” Answers

with 2 comments

[tweetmeme source=”anotherangle360″]

For those of you who might attend the next SIOP (Society for Industrial and Organizational Psychology) Conference in Chicago in April, I am pleased to note that we have been accepted to conduct a panel consisting of contributors to The Handbook of Multisource Feedback, which is approaching its 10th anniversary of publication. The panel is titled, “How has 360 degree Feedback evolved over the last 10 years?”  Panel members include Allan Church, Carol Timmreck, Janine Waclawski, David Peterson, James Farr, Manny London, Bob Jako and myself.

We received a number of thoughtful, useful comments and suggestions from the reviewers of the proposal, one of which stated this:

I would like to see a serious discussion of whether or not 360 is a singular practice. It seems as though 360 can be used with so many different interventions (succession, development, training needs analysis, supplement to coaching, …the list is HUGE) that when we say something like “is 360 legal” it is almost impossible to answer without many caveats regarding the details of the specific 360 process that was used. It’s almost as though we need to move on from ‘is 360 xyz’ to ‘if we do 360 this way, we get these outcomes and if we do 360 that way we get those outcomes.’ Can’t wait to hear the panel, this is much needed.

This is an extremely insightful observation. I have broached this topic in earlier blogs regarding alignment of purpose and decisions in design and implementation.  But there are some things that are required regardless of purpose.

To look at extremes, we might consider 360 processes where the N=1, i.e., where a single leader is given the opportunity to get developmental feedback. This is often in preparation for an experience such as a leadership development/training program, or some development program (e.g., high potential).  In these instances, it is an ad hoc process where an off-the-shelf instrument may be most practical. The instrument can be lengthy since raters will only have to fill it out one time. And typically there are major resources available to the participant in the form of coaches, trainers, and/or HR partners to ensure that the feedback is interpreted and used productively.

Compare the N=1 scenario to the N>1 process. By N>1, I use shorthand to indicate 360 processes that are applied across some segment of the population, such as a function, department, or entire organization. In these cases, it becomes much more important to have a custom designed instrument that reflects unique organization requirements (competencies, behaviors) that can create system change while simultaneously defining effective leadership to raters and ratees alike. The process requires some efficiencies due to many raters being involved, and some being asked to complete multiple forms.  We also need to plan for ways to support the many ratees in their use of the feedback.

BUT, we might also say that there are some things that are so basic as to be necessary whether N=1 or N>1.  Just this week I was sent this interview of Cindy McCauley (of the Center for Creative Leadership) (http://www.groupstir.com/resources_assets/Why%20Reliability%20and%20Validity%20Matter%20in%20360%20Feedback.pdf). Many readers will already know who Cindy is; if not, suffice to say she is highly respected in our field and has deep expertise in 360 Feedback. (In fact, she contributed a chapter to the book, “Should 360 Feedback Be Used Only for Development Purposes?” that I was also involved with.) In this interview, Cindy makes some important points about basic requirements for reliability and validity that I interpret to be applicable to all 360 processes.

What really caught my attention was this statement by Cindy:

…the scores the managers receive back mean a lot to them. They take them very seriously and are asked

to make decisions and development plans based on those scores. So you want to be sure that you can

rely on those scores, that they’re consistent and reflect some kind of accuracy.

I take the liberty (which Cindy would probably not) to expand the “make decisions” part of this statement to apply more broadly, that others (such as the leader’s manager) also use the feedback to make decisions. When she says that managers make decisions on their feedback, what decisions can they make without the support of the organization (in the person of their boss, most typically)? This is basically the crux of my argument that there is no such thing as “development only” processes. Development requires decisions and the commitment of organization resources. This only reinforces her point about the importance of validity and reliable measurement.

So what’s my point? My point is that I believe that too many ad hoc (N=1) 360 processes fall short of meeting these requirements for validity and reliability. Another debate for another time is whether off-the-shelf instruments have sufficient validity to measure unique organization requirements.  I do believe it is accurate to say that reliable measurement is often neglected in ad hoc processes when decisions are made about number of raters and quality of ratings.

For example, research indicates that raters have different “agendas” and that subordinates are the least reliable feedback providers, followed by peers and then managers. Lack of reliability can be combated in at least two ways: rater training and number of raters. We can put aside rater training (beyond having good instructions); it rarely happens despite its power and utility.

So we can improve reliability with numbers. In fact, this is really why 360 data is superior to traditional, single source evaluations (i.e., performance appraisals).  For N>1 processes, I STRONGLY recommend that all direct reports (subordinates) participate as raters. This has multiple benefits, including beefing up the number of raters for the most unreliable rater group. Then, for peers, aiming for 5-7 respondents is recommended.

My contention is that the majority of ad hoc (N=1) processes do not adhere to those guidelines. (I have no data to support that assertion, just observation.)  The problem of unreliable data due to inadequate number of raters is compounded by the fact that the decisions resulting from that flawed data are magnified due to the senior level of the leaders and the considerable organization resources devoted to their development.

When I started writing this blog, I was thinking of the title, “There is No “Right Answer,” meaning that decisions need to fit the purpose. But actually there are some “Right Answers” that apply regardless of purpose. Don’t let the “development only” argument lead to implementation decisions that reduce the reliability and validity of the feedback. In fact, many guidelines should apply to all 360 processes, whether N=1 or N>1.

©2011 David W. Bracken