What are “Strategic 360’s”?

[tweetmeme source=”strategic360s”]

A colleague recently asked me, “Exactly what is ‘Strategic 360 Feedback’?”  Heck, it’s only the name of this blog and in the name the consortium I have helped form, The Strategic 360 Forum (that is meeting for its 5^th time in April).  The concepts are also laid out pretty well in the article Dale Rose and I published in 2011 in the Journal of Business in Psychology (“When Does 360-degree Feedback Create Behavior Change? And How Would We Know It When It Does?”).

In as succinct way as I can muster, here are the four core requirements for “strategic” 360 feedback systems:

1. The content must be derived from the organization’s strategy and values, which are unique to that organization. Often derived from the organization’s values, they can be explicit (the ones that hang on the wall) or implicit (which some people call “culture”). To me, “strategic” and “off-the-shelf” is an oxymoron and the two words cannot be used in the same sentence (though I just did).
2. Participation must be inclusive, i.e., a census of the leaders/managers in the organizational unit (e.g., total company, division, location, function, level). I say “leaders/managers” because a true 360 requires that subordinates are a rater group. One reason for this requirement is that I (and many others) believe 360’s, under the right circumstances, can be used to make personnel decisions and that usually requires comparing individuals, which, in turn, requires that everyone have available the same data. This requirement also enables us to use Strategic 360’s to create organizational change, as in “large scale change occurs when a lot of people change just a little.”
3. The process must be designed and implemented in such a way that the results are sufficiently reliable (we have already established content validity in requirement #1) that we can use them to make decisions about the leaders (as in #4). This is not an easy goal to achieve, even though benchmark studies continue to indicate that 360’s are the most commonly used form of assessment in both public and private sectors.
4. The results of Strategic 360’s are integrated with important talent management and development processes, such as leadership development and training, performance management, staffing (internal movement), succession planning, and high potential processes. Research indicates that properly implemented 360 results can not only more reliable (in a statistical meaning) than single-source ratings, but are also more fair to minorities, women, and older workers. Integration into HR systems also brings with it accountability, whether driven by the process or internally (self) driven because the leader knows that the results matter.

Let me hasten to say that a) all 360’s, strategic or not, should have a development focus, and b) none of this minimizes the value of 360 processes that are used in support of the development of leaders, one at a time. There is no question that innumerable leaders have benefitted from the awareness created by feedback, though often also supported by a coach who not only helps manage the use of the feedback, but also should be creating accountability for the constructive use of the feedback.

Strategic 360 processes and “development only” processes can successfully coexist in a single organization. But they have different purposes, and purpose should be the primary driver of all design and implementation decisions.

It’s Human Nature

[tweetmeme source=”anotherangle360″]

One question that has been at the core of best practices in 360 Feedback since its inception relates to the conditions that are most likely to create sustained behavior change (at least for those of us that believe that behavior change is the ultimate goal).  Many of us believe that behavior change is not a question of ability to change but primarily one of motivation. Motivation often begins with the creation of awareness that some change is necessary, the accepting the feedback, and then moving on to implementing the change.

One of the more interesting examples of creating behavior change began when seat belts were included as standard equipment in all passenger vehicles in 1964.  I am old enough to remember when that happened and started driving not long thereafter. So using a seat belt was part of the driver education routine since I began driving and has not been a big deal for me.

The reasons for noncompliance with seatbelt usage are as varied as human nature. Some people see it as a civil rights issue, as in, “No one is going to tell me what to do.” There is also the notion that it protects against a low probability event, as in “It won’t happen to me. I’m a careful driver.” Living in Nebraska for a while, I learned that people growing up on a farm don’t “have the time” to buckle and unbuckle seatbelts in their trucks when they are learning to drive, so they don’t get into that habit. (I also found, to my annoyance, that they also never learned how to use turn signals.)

I remember back in the ‘60’s reading about a woman who wrote a car manufacturer to ask that they make the seat belts thinner because they were uncomfortable to sit on.  Really.

Some people have internal motivation to comply, which can also be due to multiple factors such as personality, demographics, training, norms (e.g., parental modeling), and so on. This is also true when we are trying to create behavior change in leaders, but we will see that these factors are not primary determinants of compliance..

In thinking about seatbelt usage as a challenge in creating behavior change, I found study from 2008 by the Department of Transportation. It is titled “How States Achieve High Seat Belt Use Rates” (DOT HS 810 962).  (Note: This is a 170 page report with lots of tables and statistical analyses, and if any of you geeks want a copy, let me know.)

The major finding of this in-depth study states:

The statistical analyses suggest that the most important difference between the high and low seat belt use States is enforcement, not demographics or funds spent on media.

This chart Seatbelt Usage in US, amongst the many in this report, seems to capture the messages fairly well to support their assertion.  This chart plots seat belt usage by state, where we see a large spread ranging from just over 60% (Mississippi) to about 95% (Hawaii).  It also shows whether each state has primary seatbelt laws (where seatbelt usage is a violation by itself), or secondary laws (where seatbelt usage can only be enforced if the driver is stopped for another purpose). Based on this table alone, one might argue causality but the study systematically shows that this data, along with others relating to law enforcement practices, are the best predictors of seatbelt usage.

One way of looking at this study is to view law enforcement as a form of external accountability, i.e., having consequences for your actions (or lack thereof). The primary versus secondary law factor largely shifts the probabilities of being caught, with the apparent desired effect on seatbelt usage.

So, back to 360 Feedback. I always have been, and continue to be, mystified as to how some implementers of 360 feedback processes believe that sustainable behavior change is going to occur in the vast majority of leaders without some form of external accountability. Processes that are supposedly “development only” (i.e., have no consequences) should not be expected to create change. In those processes, participants are often not required to, or even discouraged from, sharing their results with others, especially their manager. I have called these processes “parlor games” in the past because they are kind of fun, are all about “me,” and have no consequences.

How can we create external accountability in 360 processes?  I believe that the most constructive way to create both motivation and alignment (ensuring behavior change is in synch with organizational needs/values) is to integrate the 360 feedback into Human Resource processes, such as leadership development, succession planning, high potential programs, staffing decisions, and performance management.  All these uses involve some form of decision making that affects the individual (and the organization), which puts pressure on the 360 data to be reliable and valid. Note also that I include leadership development in this list as a form of decision making because it does affect the employee’s career as well as the investment (or not) of organization resources.

But external accountability can be created by other, more subtle ways as well. We all know from our kept and (more typically) unkept New Year’s resolutions about the power of going public with our commitments to change. Sharing your results and actions with your manager has many benefits, but can cause real and perceived unfairness if some people are doing it and others not. Discussing your results with your raters and engaging them in your development plans has multiple benefits.

Another source of accountability can (and should) come from your coach, if you are fortunate enough to have one.  I have always believed that the finding in the Smither et al (2005) meta-analysis that the presence of a coach is one determinant of whether behavior change is observed is due to the accountability that coaches create by requiring the coachee to specifically state what they are going to do and to check back that the coachee has followed through on that commitment.

Over and over, we see evidence that, when human beings are not held accountable, more often than not they will stray from what is in their best interests and/or the interests of the group (organization, country, etc.).  Whether it’s irrational (ignoring facts) or overly rational (finding ways to “get around” the system), we should not expect that people will do what is needed, and we should not rely on our friends, neighbors, peers or leaders to always do what is right if there are no consequences for inaction or bad behavior.

©2012 David W. Bracken

I Need an Exorcism

[tweetmeme source=”anotherangle360″]

Being the 360 Feedback nerd I am, I love it when some new folks get active on the LinkedIn 360 discussion group. One discussion emerged recently that caught my eye, and I have been watching it with interest, mulling over the perspectives and knowing I had to get my two cents in at some point.

Here is the question:

How many raters are too many raters?

We normally recommend 20 as a soft limit. With too many, we find the feedback gets diluted and you have too many people that don’t work closely enough with you to provide good feedback. I’d be curious if there are any suggestions for exceptions.

This is an important decision amongst the dozens that need to be made in the course of designing and implementing 360 processes. The question motivated me to pull out The Handbook of Multisource Feedback and find the excellent chapter on this topic by James Farr and Daniel Newman (2001), which reminded me of the complexity of this decision. Let me also reiterate that this is another decision that has different implications for “N=1” 360 processes (i.e., feedback for a single leader on an ad hoc basis) versus “N>1” systems (i.e., feedback for a group of participants); this blog and discussion is focused on the latter.

Usually people argue that too many surveys will cause disruption in the organization and unnecessary “soft costs” (i.e., time). The author of this question poses a different argument for limiting the rater population, which he calls “dilution” due to inviting unknowledgeable raters.  For me, one of the givens of any 360 system is that the raters must have sufficient experience with the ratee to give reliable feedback. One operationalization of that concept is to require that an employee must have worked with/for the ratee for some minimum amount of time (e.g., 6 months or even 1 year), even if he/she is a direct report. Having the ratee select the raters (with manager approval) is another practice that is designed to help get quality raters that then also facilitate the acceptance of the feedback by the ratee. So “dilution” due to unfamiliarity can be combated with that requirement, at least to some extent.

One respondent to this question offers this perspective:

The number of raters depends on the number of people that deal with this individual through important business interactions and can pass valuable feedback based on real experience. There is no one set answer.

I agree with that statement. Though, while there is no one set answer, some answers are better than others (see below).

In contrast, someone else states:

We have found effective to use minimum 3 and maximum 5 for any one rater category.

The minimum of 3 is standard practice these days as a “necessary but not sufficient” answer to the number of raters. As for the maximum of 5, this is also not uncommon but seems to ignore the science that supports larger numbers.  When clients seek my advice on this question of number of raters, I am swayed by the research published by Greguras and Robie (1998) who collected and researched the question of the reliability of various rater sources (i.e., subordinates, peers and managers). They came to the conclusion that different rater groups provide differing levels of reliable feedback, probably because the number of “agendas” lurking within the various types of raters. The least reliable are the subordinates, followed by the peers, and then the managers, the most reliable rater group.

One way to address rater unreliability is to increase the size of the group (another might be rater training, for example). Usually there is only one manager and best practice is to invite all direct reports (who meet the tenure guidelines), so the main question is the number of peers. This research suggests that 7-9 is where we need to aim, noting also that that is the number of returns needed, so inviting more is probably a good idea if you expect less than a 100% response rate.

Another potential rater group is external customers. Recently I was invited to participate in a forum convened by the American Board of Internal Medicine (ABIM) to discuss the use of multisource feedback in physician recertification processes. ABIM is one of 24 member Boards of the American Board of Medical Specialties (ABMS), which has directed that some sort of multisource (or 360) feedback be integrated into recertification.

The participants in this forum included many knowledgable, interesting researchers on the use of 360 in the context of medicine (a whole new world for me, which was very energizing). I was invited to represent the industry (“outside) perspective. One of the presenters spoke to the challenge of collecting input from their customers (i.e., patients), a requirement for them. She offered up the number of 25 as the number of patients needed to create a reliable result, using very similar rationale as Greguras and Robie regarding the many individual agendas of raters.

Back to LinkedIn, there was then this opinion:

I agree that having too many raters in any one rater group does dilute the feedback and make it much harder to see subtleties. There is also a risk that too many raters may ‘drown out’ key feedback.

This is when my head started spinning like Linda Blair in The Exorcist.  This perspective is SO contrary to my 25 years of experience in this field that I had to prevent myself from discounting it as my head continued to rotate.  I have often said that a good day for me includes times when I have said, “Gee, I have never thought of (insert topic) in that way.” I really do like hearing new and different views, but it’s difficult when they challenge some foundational belief.

For me, maybe THE most central tenet of 360 Feedback is the reliance on rater anonymity in the expectation (or hope) that it will promote honesty. This goes back to the first book on 360 Feedback by Edwards and Ewen (1996) where 360’s were designed with this need for anonymity being in the forefront. That is why we use the artificial form of communication of using anonymous questionnaires and usually don’t report in groups of less than 3. We know that violations of the anonymity promise result in less honesty and reduced response rates, with the grapevine (and/or social media) spreading violated trust throughout the organization.

The notion that too many raters will “drown out key feedback” seems to me to be a total reversal of this philosophy of protecting anonymity. It also seems to place an incredible amount of emphasis on the report itself where the numbers become the sole source of insight. Other blog entries of mine have proposed that the report is just the conversation starter, and that true insight is achieved in the post-survey discussions with raters and manager.

I recall that in past articles (see Bracken, Timmreck, Fleenor and Summers, 2001) we made the point that every decision requires what should be a conscious value judgment as to who the most important “customer” is for that decision, whether it be the rater, ratee, or the organization. For example, limiting the number of raters to a small number (e.g., 5 per group or not all Direct Reports) indicates that the raters and organization are more important than the ratee, that is, that we believe it is more important to minimize the time required of raters than it is to provide reliable feedback for the ratee. In most cases, my values cause me to lobby on behalf of the ratee as the most important customer in design decisions.  The time that I will rally to the defense of the rater as the most important customer in a decision is when anonymity (again, real or perceived) is threatened. And I see these arguments for creating more “insight” by keeping rater groups small or subdivided are misguided IF these practitioners share the common belief that anonymity is critical.

Finally (yes, it’s time to wrap this up), Larry Cipolla, an extremely experienced and respected practitioner in this field, offers some sage advice with some comments, including the folly of increasing rater group size by combining rater groups. As he says, that is pure folly. But I do take issue with one of his practices:

We recommend including all 10 raters (or whatever the n-count is) and have the participant create two groups–Direct Reports A and Direct Reports B.

This seems to me to be a variation on the theme of breaking out groups and reducing group size with the risk of creating suspicions and problems with perceived (or real) anonymity. Larry, you need to show that doing this kind of subdividing creates higher reliability in a statistical sense that can overcome the threats to reliability created by using smaller N’s.

Someone please stop my head from spinning. Do I just need to get over this fixation with anonymity in 360 processes?

References

Bracken, D.W., Timmreck, C.W., and Church, A.H. (2001). The Handbook of Multisource Feedback. San Francisco: Jossey-Bass.

Bracken, D.W., Timmreck, C.W., Fleenor, J.W., and Summers, L. (2001). 360 feedback from another angle. Human Resource Management, 1, 3-20.

Edwards, M. R., and Ewen, A.J.  (1996). 360° Feedback: The powerful new model for employee assessment and performance improvement. New York: AMACOM.

Farr, J.L., and Newman, D.A. (2001). Rater selection: Sources of feedback. In Bracken, D.W., Timmreck, C.W., and Church, A.H. (eds.), The Handbook of Multisource Feedback. San Francisco: Jossey-Bass.

Greguras, G.J., and Robie, C. (1998).  A new look at within-source interrater reliability of 360-degree feedback ratings. Journal of Applied Psychology, 83, 960-968.

©2012 David W. Bracken

What is a “5” ?

[tweetmeme source=”anotherangle360″]

The Sunday New York Times Business section typically has a feature called “Corner Office” where a CEO is interviewed. These CEO’s seem to often be from small businesses. The one today (October 16, 2011), for example, is the CEO of a 25-person, $6 million consulting firm. The questions are often the same, having to do with the process of becoming a leader, lessons learned, hiring and promotion strategies. I have referenced these in a couple earlier blogs since they touch on topics relative to behavior change, leadership development and culture change that are relevant to aspects of 360 processes.

In today’s column, the CEO was asked about how he hires. He notes that part of the interview often includes asking the applicant to rate him/herself on a five point scale regarding various areas of knowledge and experience from their resume. If the applicant rates themselves as a “5,” he then asks if there is nothing else that they could learn about whatever it is. Of course, they say, “Oh, no, no. There is.” To which the CEO asks, “then why did you rate yourself a five?”  And he goes on to say he has never hired someone who rates themself as a five.

While this CEO’s decision not to hire these high self raters may seem arbitrary, think of the culture he is trying to create by the people he selects and the message this communicates to new hires during the process. He says that he appreciates humbleness and their understanding that they don’t know everything.  (In an earlier column, the CEO asks applicants if they are “nice,” and then explains what “nice” means in their culture, i.e., a good team player.)

(Someone told me of a senior executive who greeted applicants in the lobby and asked them whether they should take the stairs or elevator. If they said elevator, he didn’t hire them. That seems less job related, but is our “5” CEO doing a similar thing? Food for thought.)

We don’t use 360’s to hire people (though I have heard of multisource reference checks  from  past coworkers that are being positioned as 360’s), but we do have an opportunity with 360’s to create or support a culture when they involve many people. But we also know that 360’s are notorious for having severe leniency, i.e., mostly 4’s and 5’s on a five point scale.

Do all these 5’s that we collect mean that our leaders can’t get any better at what they do? Of course not. But that seems to be the message that we allow and even reward (even if not tangibly).

The vast majority of 360 processes use an Agree/Disagree (Likert) scale where “Strongly Agree” is scored as a “5” (scales that score it as a “1” seem confusing and counterintuitive to me).  The VAST majority of processes also do not include rater training that could be used to help raters (and ratees for that matter) to stop attaching any meaning to “Strongly Agree” that they wish. Which they currently do.

I have used a rating scale where “5” is defined as “role model, in the top 5-10%” that attempts to create a frame of reference for raters (and ratees) that does help reduce leniency effects.

What if we defined “5” as “can’t get any better” or something equivalent to that. I think “role model” implies that this person can be a teacher as well as example to others, and perhaps doesn’t need to get better (i.e., focus on other areas of improvement).  Some raters will undoubtedly ignore those directions, but rater training can help drill in the need for everyone to reconfigure their conceptualization of what optimal behavior is and, by the way, foster the learning and development culture that our CEO seems to be nurturing.

A recalibration of rating scales is badly needed in this field. We need to stop raters from giving all “5’s” and from ratees giving self ratings of all “5’s”.  With our current mentality on rating scales, there is really nothing to stop rating inflation. It should be no surprise that senior leaders find it to be difficult to use and support our 360 programs.

©2011 David W. Bracken

So Now What?

[tweetmeme source=”anotherangle360″]

This is the one year anniversary of this blog. This is the 44^th post.  We have had 2,026 views, though the biggest day was the first with 38 views.  I have had fewer comments than I had hoped (only 30), though some LinkedIn discussion have resulted. Here is my question: Where to go from here? Are there topics that are of interest to readers?

Meanwhile, here is my pet peeve(s) of the week/month/year:  I was recently having an exchange with colleagues regarding a 360 topic on my personal Gmail account and up pops ads on the margin for various 360 vendors (which is interesting in itself), the first of which is from Qualtrics (www.qualtrics.com) with the heading, “Create 360s in Minutes.”

The topic of technology run amok has been covered before here (When Computers Go Too Far, http://wp.me/p10Xjf-3G), my peevery was piqued (piqued peevery?) when I explored their website and saw this claim:  USE VALIDATED QUESTIONS, FORMS and REPORTS.”

What the heck does that mean?  What are “validated” forms and reports, for starters?

The bigger question is, what is “validity” in a 360 process?  Colleagues and I (Bracken, Timmreck, Fleenor and Summers, 2001; contact me if you want a copy) have offered up a definition of validity for 360’s that holds that it consists of creating sustainable change in behaviors valued by the organization.  Reliable items, user friendly forms and sensible reports certainly help to achieve that goal, but certainly cannot be said to be “valid” as standalone steps in the process.

The Qualtrics people don’t share much about who they are. Evidently their founder is named Scott and teaches MBA’s.  They appear to have a successful enterprise, so kudos!  I would like to know how technology vendors claim to have “valid” tools and what definition of validity they are using.

Hey maybe I will get my 31^st comment?

©2011 David W. Bracken

I Don’t Care

[tweetmeme source=”anotherangle360″]

Last week I led a workshop for the Personnel Testing Council of Metropolitan Washington that was a modified reprise of the workshop Carol Jenkins and I did at the Society for Industrial and Organizational Psychology in April. I really enjoy these workshops and the opportunity to interact face-to-face with practitioners in the field of 360 degree feedback.

I do wish that participants in these workshops would engage me in a little more debate, and, to that end, I sometimes throw out comments in the hope of raising some hackles. For example, at the PTCMW session, I twice said “I don’t care” regarding two topics that I will explain below. Unfortunately, no one took the bait in the workshop, but maybe I can lure some of you into the discussion using this blog as a vehicle.

So here are the two areas where a ton of research is being done but where, as a practitioner, I don’t care:

1)      The personality of the participant. I don’t care. Everyone seems to want to know how the personality of the participant is going to affect his/her reaction to the feedback.  In past blogs, I have fessed up to being a behaviorist, and in that respect all I really “care” about is getting the person to accept the feedback and to change, whether they want to or not. In my last blog, I used the examples of people’s apparent reluctance to do simple things like apologize for mistake and/or to say “thank you.”  Behaviorally, those are pretty easy things to do, but evidently some internal force (e.g., personality) makes it difficult.  In fact, those internal forces vary greatly across people, and I find chasing them down to not be a very fruitful use of time for the participant or for myself. If the organization and feedback tells you that you need to modify your behavior, just do it!

Sometimes what is going on inside the person’s head is more an issue of awareness than of personality, and awareness is something we can change through 360’s. Occasionally the journey from awareness to acceptance is difficult due to personality factors. It is our job to design the 360 process to make it difficult to not accept the feedback, including ensuring that raters are knowledgeable, reliable, motivated and in sufficient quantity.

On a practical level, when many 360 processes involve dozens or hundreds of participants, it becomes very challenging to integrate personality assessment, for example, into the mix. Not to say it can’t be done. Carol Jenkins does some of that in her practice with groups of feedback recipients. But part of my “I don’t care” mentality has come from a need to get large numbers of people to use the feedback productively without being able to “get inside their head.”

2)      The gap between self-ratings and “other” ratings. I don’t care. As a psychologist, I do find it interesting to see how ratees approach self-ratings, especially the first time around. And they usually change their self-ratings once they see how they are perceived by others. But I am increasingly convinced that self-ratings are more a reflection of the ratee’s agenda than any real self-assessment. (All raters are susceptible to using their ratings to this kind of error.) One memorable instance for me was in working with a Chief Legal Officer who gave himself all 5’s and stated, “do you think I would be crazy enough to actually document less than optimal performance?”

I DO think that participants should complete the rating process, but for other reasons. One is to ensure that they are familiar with the content and how he/she is expected to behave as defined by the organization. Secondly, it is some evidence of at least minimal commitment to the process.

In general, I am not very interested in why a ratee behaves in a certain way if it needs to change. It is highly unlikely that we can change the “why” part of behavior (i.e., personality) other than to affect their awareness of how they are perceived and the importance of accepting that feedback on the way to behaving differently.What is going on in the person’s head is fun for psychologists to research, but doesn’t necessarily help achieve sustainable behavior change.

©2011 David W. Bracken

On the Road… and Web and Print

[tweetmeme source=”anotherangle360″]

I have a few events coming up in the next 3 weeks or so that I would like to bring to your collective attention in case you have some interest.  One is free, two are not (though I receive no remuneration). I also have an article out that I co-authored on 360 feedback.

In chronological order, on May 25 Allan Church, VP Global Talent Development at PepsiCo, and I will lead a seminar titled, “Integrating 360 & Upward Feedback into Performance and Rewards Systems” at the 2011 World at Work Conference in San Diego (www.worldatwork.org/sandiego2011).  I will be offering some general observations on the appropriateness, challenges, and potential benefits of using 360 Feedback for decision making, such as performance management. The audience will be very interested in Allan’s descriptions of his experiences with past and current processes that have used 360 and Upward Feedback for both developmental and decision making purposes.

On June 8, I am looking forward to conducting a half day workshop for the Personnel Testing Council of Metropolitan Washington (PTCMW) in Arlington, VA, titled “360-Degree Assessments: Make the Right Decisions and Create Sustainable Change” (contact Training.PTCMW@GMAIL.COM or go to WWW.PTCMW.ORG). This workshop is open to the public and costs $50.  I will be building from the workshop Carol Jenkins and I conducted at The Society for Industrial and Organizational Psychology. That said, the word “assessments” in the title is a foreshadowing of a greater emphasis on the use of 360 Feedback in a decision making context and an audience that is expected to have great interest in the questions of validity and measurement.

On the following day, June 9 (at 3:30 PM EDT), I will be part of an online virtual conference organized by the Institute of Human Resources and hr.com on performance management. My webinar is titled, “Using 360 Feedback in Performance Management: The Debate and Decisions,” where the “decisions” part has multiple meanings. Given the earlier two sessions I described, it should be clear that I am a proponent of using 360/Upward Feedback for decision making under the right conditions. The other take on “decisions” is the multitude of decisions that are required to create those “right conditions” in the design and implementation of a multisource process.

On that note, I am proud to say that Dale Rose and I have a new article in the Journal of Business and Psychology (June) titled, “When does 360-degree feedback create behavior change? And how would we know it when it does?” Our effort is largely an attempt to identify the critical design factors in creating 360 processes and the associated research needs.

This article is part of a special research issue (http://springerlink.com/content/w44772764751/) of JBP and you will have to pay for a copy unless you have a subscription. As a tease, here is the abstract:

360-degree feedback has great promise as a method for creating both behavior change and organization change, yet research demonstrating results to this effect has been mixed. The mixed results are, at least in part, because of the high degree of variation in design features across 360 processes. We identify four characteristics of a 360 process that are required to successfully create organization change, (1) relevant content, (2) credible data, (3) accountability, and (4) census participation, and cite the important research issues in each of those areas relative to design decisions. In addition, when behavior change is created, the data must be sufficiently reliable to detect it, and we highlight current and needed research in the measurement domain, using response scale research as a prime example.

Hope something here catches your eye/ear!

©2011 David W. Bracken

The Death Card

[tweetmeme source=”anotherangle360″]

A number of (pre-recession) years ago, I belonged to a firm that was operating in the black and held some very nice off-site meetings for its consultants. At one such event, we had an evening reception that had some fun activities, one of which being a Tarot reader. I don’t even read horoscopes but there was no one waiting and I decided to give it a try (the first and last time).  I obviously didn’t know much about Tarot but it seemed like the last card to be turned over was the most important. And, lo and behold, it was the Death card! I remember a pause from the Reader (perhaps an intake of breath?), and then a rapid clearing of the cards with some comment to the effect of, “That’s not important.”  Session over.

Well, I guess the good news is that I am still here (most people would agree with that I think).  My purpose for bringing this up is not to discuss superstitions and the occult, but to reflect on how people react to and use 360 feedback.

In fact, I have been known to call some 360 processes “parlor games, “which relates directly to my Tarot experience. That was a true “parlor game.”  What is a parlor game? My definition, for this context, is an activity that is fun and has no consequences, where a person can be the focus of attention with low risk of embarrassment and effort.  Since I strongly believe in self determination, I do my best to not let arbitrary events that I cannot control to affect my life. That would include a turn of a card, for starters.

So how do we ensure that 360 Feedback isn’t a parlor game and does matter? I propose that two important factors are Acceptance and Accountability.

Some of the design factors that promote Acceptance would include:

• Use a custom instrument (to create relevance)
• Have the rater select raters, with manager approval (to enhance credibility of feedback)
• Enhance rater honesty and reliability (to help credibility of data)
• Invite enough raters to enhance reliability and minimize effects of outliers
• Be totally transparent to purpose, goals, and use (not mystical, magic, inconsistent or arbitrary)

Factors that can help create Accountability (and increase the probability of behavior change) include:

• Require leaders to discuss results and development plans with raters (like going public with a New Year’s Resolution)
• Include results as a component of performance management, typically in the development planning section, to create consequences for follow through, or lack thereof
• Ensure that the leader’s manager is also held accountable for properly using results in managing and coaching
• Conduct follow-up measures such as mini-360’s and/or annual readministrations.

Some 360 processes appear to define success as just creating awareness in the participants, hoping that the leader will be self motivated to change. That does happen; some leaders do change, at least for a while, and maybe even in the right way. (Some people probably change based on Tarot readings too!).  For those leaders who need to change the most, it usually doesn’t happen without Acceptance and Accountability.

Simply giving a feedback report to a leader and stopping there seems like a parlor game to me. A very expensive one.

©2011 David W. Bracken

Maybe Purpose Doesn’t Matter?

[tweetmeme source=”anotherangle360″]

While there are many discussions and debates within the 360 Feedback community (including one regarding randomizing items currently on LinkedIn that I will address in a later blog), probably none is more intense and enduring than the issue of the proper use of 360 results. In The Handbook of MultiSource Feedback, a whole chapter (by Manny London) was dedicated to “The Great Debate” regarding using 360 for developmental vs. decision making purposes. In fact, in the late 90’s an entire book was published by the Center for Creative Leadership based on a debate I organized at SIOP.

I have argued in earlier blogs and other forums that I believe this “either/or” choice is a false one for many reasons. For example, even “development only” uses require decisions that affect personal and organizational outcomes and resources. Also, even when used for decision (including succession planning, staffing, promotions, and, yes, performance management), there is always a development component.

One of the aggravating blanket statements that is used by the “development only” crowd is that respondents will not be honest if they believe that the results will be used to make decisions that might be detrimental to the ratee, resulting in inflated scores with less variability. I would say that, in fact, that is by far the most common argument for the “development only” proponents, and one that is indeed supported by some research studies.

I have just become aware of an article published 3 years ago in the Journal of Business and Psychology (JBP) relating to multisource feedback, titled “Factors Influencing Employee Intentions to Provide Honest Upward Feedback Ratings” (Smith and Fortunato, 2008).  For those of you who are not familiar with JBP, it is a refereed journal of high quality that should be on your radar and, in full disclosure, a journal for which I am an occasional reviewer.

The study was conducted at a behavioral health center with a final sample of 203 respondents. The employees filled out a questionnaire about various aspects of an upward feedback process that was being implemented in the future.

The article is fairly technical and targeted toward the industrial/organizational community. I have pulled out one figure for the geeks in the audience to consume if desired (click on “360 Figure”) . But let me summarize the findings of the study.

The outcome (dependent variable) that was of primary interest to the researchers is foreshadowed in the title, i.e., what factors lead to intentions to respond honestly in ratings of a supervisor (upward feedback).  The most surprising result (as highlighted in the discussion by the authors) was that purpose (administrative versus developmental) had no predictive value at all! Of all the predictor variables measured, it was the least influential with no practical (statistical) significance.

What does predict intentions to provide honest feedback? One major predictor is the level of cynacism, with (as you might guess) cynical attitudes resulting in less honesty. The study suggests that cynical employees fear retaliation by supervisors and are less likely to believe that the stated purpose will be followed. The authors suggest that support and visible participation by senior leaders might help reduce these negative attitudes. We also need to continue to protect both real and perceived confidentiality, and to have processes to identify cases of retaliation and hold the offending parties accountable.

The other major factor is what I would label as rater self confidence in their ability as a feedback provider. Raters need to feel that their input is appropriate and valued, and that they know how the process will work. They also have a need to feel that they have sufficient opportunity to observe.  The authors appropriately point to the usefulness of rater training to help accomplish these outcomes. They do not mention the rater selection process as being an important determinant of opportunity to observe, but that is obviously a major factor in ensuring that the best raters are chosen.

One suggestion the authors make (seemingly out of context) that is purported to help improve the honesty of the feedback is to use reverse-worded items to keep raters from choosing only socially desirable responses (e.g., Strongly Agree).  I totally disagree with practices such as reverse wording and randomization which may actually reduce the reliability of the instrument (unless the purpose is for research only). For example, at our SIOP Workshop, Carol Jenkins and I will be showing an actual 360 report that uses both of those methods (reverse wording and randomization). In this report (that Carol had to try to interpret for a client), the manager (“boss”) of the ratee had give the same response (Agree) to two versions of the same item where one was reverse scored. In other words, the Manager was Agreeing that the ratee was both doing and not doing the same thing.

Now what? The authors of this study seem to suggest that situations like this would invalidate the input of this manager, arguably the most important rater of all.  Now we could just contact the manager and try to clarify his/her input. But the only reason we know of this situation is that the manager is not anonymous (and they know that going into the rating process). If this same problem of rating inconsistency occurs with other rater groups, it is almost impossible to rectify since the raters are anonymous and confidential (hopefully).

This is only one study, though a well designed and analyzed study in a respected journal. I will not say that this study proves that purpose does not have an effect on honesty. Nor should anyone say that other studies prove that purpose does affect honesty. To be clear, I have always said that it may be appropriate to use 360 results in decision making under the right conditions, conditions that are admittedly often difficult to achieve. This is in contrast to some practitioners who contend that it is never appropriate to do so, under any conditions.

Someday when I address the subject of organizational readiness, I will recall the survey used in this research which was administered in anticipation of implementing an upward feedback process. This brief (31 item) survey used for this study would be a great tool to assess readiness in all 360 systems.

One contribution of this research is to point out that intention to be honest is as much a characteristic of the process as it is of the person. Honesty is a changeable behavior in this context through training, communication, and practice. Making blanket statements about rater behavior and how a 360 program should or shouldn’t be used are not productive.

360 Figure

©2011 David W. Bracken

Not Funny

[tweetmeme source=”anotherangle360″]

I seem to be in a bit of a rut with themes around humor and now commercials. Despite trying to bypass as many commercials as possible with my DVR, occasionally I do see one and sometimes even for the better.

One that caught my eye/ear is one by IBM that starts with a snippet of a Groucho Marx (whom I also like very much) where he states, “This morning I shot an elephant in my pajamas.”  Of course, the fun part is when he follows, “How he got in my pajamas, I will never know.”  Ba bump.

The commercial goes on to talk about a computer called Watson that has been developed by IBM with capabilities that will be used to compete on Jeopardy (another favorite). The point is that language has subtle meanings, euphemisms, metaphors, nuances and unexpected twists that are difficult for machines to correctly comprehend.

In the context of 360 Feedback, the problem is that we humans are sometimes not so good at picking up the subtleties of language as well. We need to do everything we can to remove ambiguity in our survey content, acknowledging that we can never be 100% successful.

We have all learned, sometimes the hard way, about how our attempts to communicate with others. How often have we had to come to grips with how our seemingly clear directions have been misunderstood by others?

I became sensitized to this question of ambiguity in language during the quality movement of the 80’s and the work of Peter Senge as embodied in The Fifth Discipline and the accompanying Fifth Discipline Fieldbook. (Writing this blog has spurred me to pull out this book; if you youngsters are not aware of Senge’s writings, it is still worth digging out. There is a 2006 Edition which I confess I have not read yet.)

There are many lessons in these books regarding the need to raise awareness about our natural tendencies as humans to fall back on assumptions, beliefs, values, etc., often unconsciously, in making decisions, trying to influence, and taking actions. One lesson that has particularly stuck with me in the context of 360’s is the concept of mental models, which Senge defines as, “deeply ingrained assumptions, generalizations, or even pictures or images that influence how we understand the world and how we take action.”  In the Fieldbook, he uses an example of the word “chair” and how that simple word will conjure up vastly different mental images of what a “chair” is, from very austere, simple seats to very lush, padded recliners and beyond. (In fact, it might even create an image of someone running a meeting if we are to take it even farther.)

So Groucho created a “mental model” (or assumed one) of us visualizing him in his pajamas with a gun chasing an elephant. Then he smashes that “assumption” we made by telling us that the elephant was wearing the pajamas. That is funny in many ways.

Sometimes we are amused when we find we have made an incorrect assumption about what someone has told us. I have told the story before of the leader who made assumptions about his low score on “Listens Effectively.” He unexpectedly found that his assumptions were unfounded and the raters were simply telling him to put down his PDA. That could be amusing and also a relief since it is an easy thing to act on.

360 Feedback is a very artificial form of communication where we rely on questionnaires to allow raters to “tell” the ratee something while protecting their anonymity. This also has the potential benefit of allowing us to easily quantify the responses which, in turn, can be used to measure gaps (between rater groups, for example) and track progress over time.

Of course this artificial communication creates many opportunities for raters to misunderstand or honestly misuse the intent of the items and, in turn, for ratees to misinterpret the intended message from the raters. We need to do our best to keep language simple and direct, though we can never prevent raters applying different “mental models.”

Take an item like, “Ensures the team has adequate resources.” Not a bad question. But, like “chair,” “resources” can create all sorts of mental images such as people (staff), money (budget), equipment (e.g., computers), access to the leader, and who knows what else! We could create a different item for each type of resource if we had an unlimited item budget, which we don’t.

This potential problem is heightened if there will be multiple languages used, creating all sorts of issues with translations, cultural perspectives, language nuances, and so on.

In the spirit of “every problem has a solution,” I can think of at least four basic recommendations.

First, be diligent in item writing to keep confusion to a minimum.  For example:

• Use simple words/language
• Don’t use euphemisms (“does a good job”)
• Don’t use metaphors (“thinks outside the box”)
• Don’t use sports language (“creates benchstrength”)
• Keep all wording positive (or cluster negatively phrased items such as derailers in one dimension with clear instructions)

Second, conduct pilot tests with live raters who can give the facilitator immediate feedback on wording in terms of clarity and inferred meaning.

Third, conduct rater training. Some companies tell me that certain language is “ingrained” in their culture, such as “think outside the box.” (I really wonder how many people really know the origins of that metaphor. Look it up in Wikipedia if you don’t.)  I usually have to defer to their wishes, but still believe that their beliefs may be more aspirational than factual. Including a review of company-specific language (which does have some value in demonstrating the uniqueness of the 360 content) during rater training will have multiple benefits.

Fourth, acknowledge and communicate that it is impossible to prevent misinterpretations by the senders (raters) and the receivers (ratees). This will require that the ratee discuss results with the raters and ensure that they are all “on the same page”. (metaphor intended with tongue in cheek).

I bet that some ratees do actually laugh (or at least chuckle) if/when they hear how some raters interpret the questions.  But more typically it is not funny. And it is REALLY not funny if the ratee invests time and effort (and organizational resources) taking action on false issues due to miscommunication.

(Note: For those interested, Carol Jenkins and I will be talking about these issues in our SIOP Pre-Conference workshop on 360 Feedback on April 13 in Chicago.)

©2011 David W. Bracken