Strategic 360s

Making feedback matter

So Now What?

with 7 comments

[tweetmeme source=”anotherangle360″]

This is the one year anniversary of this blog. This is the 44th post.  We have had 2,026 views, though the biggest day was the first with 38 views.  I have had fewer comments than I had hoped (only 30), though some LinkedIn discussion have resulted. Here is my question: Where to go from here? Are there topics that are of interest to readers?

Meanwhile, here is my pet peeve(s) of the week/month/year:  I was recently having an exchange with colleagues regarding a 360 topic on my personal Gmail account and up pops ads on the margin for various 360 vendors (which is interesting in itself), the first of which is from Qualtrics (www.qualtrics.com) with the heading, “Create 360s in Minutes.”

The topic of technology run amok has been covered before here (When Computers Go Too Far, http://wp.me/p10Xjf-3G), my peevery was piqued (piqued peevery?) when I explored their website and saw this claim:  USE VALIDATED QUESTIONS, FORMS and REPORTS.”

What the heck does that mean?  What are “validated” forms and reports, for starters?

The bigger question is, what is “validity” in a 360 process?  Colleagues and I (Bracken, Timmreck, Fleenor and Summers, 2001; contact me if you want a copy) have offered up a definition of validity for 360’s that holds that it consists of creating sustainable change in behaviors valued by the organization.  Reliable items, user friendly forms and sensible reports certainly help to achieve that goal, but certainly cannot be said to be “valid” as standalone steps in the process.

The Qualtrics people don’t share much about who they are. Evidently their founder is named Scott and teaches MBA’s.  They appear to have a successful enterprise, so kudos!  I would like to know how technology vendors claim to have “valid” tools and what definition of validity they are using.

Hey maybe I will get my 31st comment?

©2011 David W. Bracken

Advertisements

7 Responses

Subscribe to comments with RSS.

  1. David,

    You’re probing around the margins of voodoo OD. I’m grateful for your peevery and dedication to bringing clarity and standards to this important subject. I suspicion that widespread abuse and misapplication of the methods has very great costs indeed to regular folks…that lots of poor decisions are getting made every day, that too much money is being tossed up into the air and scattered to the four winds.

    Keep it up!

    Carl

    August 10, 2011 at 9:40 am

  2. Hey David! I work at Qualtrics, and came across your post. Thanks for raising some great questions, and giving us an opportunity to share a little more about our use of the term “validated.”

    First off, I thought I’d provide a bit more info about us and our products. Qualtrics was founded back in 1997 with our online survey solution, called the ‘Research Suite.’ Just last year, Stanford, a long time Qualtrics customer, approached us with the suggestion that we create a 360 solution that’s as powerful and easy to use as our flagship product. We were excited by the challenge, and after some terrific collaboration and intense development, we launched Qualtrics 360™ earlier this year.

    We feel like our product is very competitive for a variety of reasons…but one particular feature that makes us stand out is our emerging 360 marketplace. Think of it like an app store, where a client can browse a selection of professionally built 360 forms and their corresponding reports, then launch one in minutes to a list of their participants.

    It’s these marketplace questions, forms and reports that fit under our intended definition of ‘validated’—namely “instruments that actually measure what they’re designed to measure.” We aim to be the best platform in the world for enabling people to create, distribute and even market their own 360s, but we readily admit we are not in the business designing the instruments themselves. So it’s not our technology that we deem validated, rather a handful of excellent instruments that our partners have created to promote the kind of sustainable organizational changes you cite in your definition.

    I hope this clarifies our position, and I apologize for any confusion it may have caused. We certainly don’t want to over-promise anything to our clients (prospective or current), and will take your post as an opportunity to re-examine what it means to offer ‘validated’ instruments.

    Also—we’d certainly love to continue the dialogue and show you what our product is capable of. If you’d be interested in checking out a quick demo, please let me know and I’d be more than happy to set it up. As an expert in this space, we’d certainly appreciate your feedback.

    Best regards,
    Tom McConnon

    Tom McConnon

    August 10, 2011 at 12:39 pm

  3. Tom

    that marketing piece is as clear as mud.

    Technology I get – I know that world.

    Key questions are the “boring” old questions of validity and business impact. So it woul;d be useful to provide decent references detailing the empirical stuff that indicates what is different about your approach and the real world impact it’s had.

    Regards

    Andrew

    A Munro

    August 10, 2011 at 5:33 pm

  4. Well put Andrew. Tom can’t be blamed for trying but I had the same reaction as you did. I’m not an academic but am intimately interested – and experienced – in the technical aspects of asking people to respond to questions in structured (dare I say standardized?) ways that can be shown to have both meaning and differential diagnostic power. It’s a game that cannot be played in short pants, as it were, the coefficients of validity and reliability etc. are the very first things I look for. And, if they are not immediately in evidence…off to the dustbin (or recycle bin as the case may be). A “demo”?!

    Cheers,

    Carl

    Carl

    August 10, 2011 at 9:17 pm

  5. Thank you all for your comments! Now that the commentary has subsided, I will offer one or two of my own.
    Tom, I really appreciate your perspectives, though clearly not everyone (including me) adopts your defense of validity; that’s largely why I was wondering (and still wondering) about the background/credentials of your team. I would still contend, for starters, that forms and reports should not be evaluated as tbeing “valid”, but more as to their reliability. Reliability is the sine qua non of validity but doesn’t equate to validity.
    Carl, my definition of “validity” doesn’t include the necessity of having predictive power (coefficients). I believe that a “valid” 360 measures relevant (important) behaviors in a reliable way, more akin to a performance appraisal than a ‘test”. To carry it further, a valid 360 will create aligned behavior change, which does require accurate reporting and interpretation for starters.

    David Bracken

    August 16, 2011 at 11:25 pm

  6. Thank you David – your clarification helped – I’ve seen the methods abused, i.e., applied in many ways “as if” there was underlying, (and defensible), predictive power. That is where I’m coming from. Your definition is perfectly fine and your stated goals well matched to it. Developmental activity is inherently a good thing…alignment, emerging from mutual dialog where operational definitions are clarified are as well.

    Issues, for me, arise when decision makers start making decisions about individuals based upon 360 “results”. Often enough, (I’m reasonably sure), non-technical, inexperienced people – any child can create a survey after all – are tasked with creating and deploying a 360 within ABC Corp.. Additionally, we see market driven expressions of the method which pay scant regard to what organizations might do with the “results” that their technologies can serve up.

    I’m not convinced I/O in general does strong enough work aimed at educating the “public” and preventing damage being visited upon people’s lives.

    Thanks again for your work in this regard.

    Carl

    August 17, 2011 at 8:27 am

  7. […] used my last blog (https://dwbracken.wordpress.com/2011/08/09/so-now-what/)  to start LinkedIn discussions in the 360 Feedback and I/O Practitioners group, asking the […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: