This is going to be more of a rant than usual posts because today I pretty much reached my limit on a sore subject:
I was tasked with taking a survey today designed to identify gaps in corporate ethics compliance. Other than the misspelling of my organization, it started off innocently enough… and then I reached a question that locked me up.
The question presented a list of six hierarchal roles and asked the survey-takers to rate them from 1 to 6 based on setting the proper tone in the workplace for ethics and integrity, with 1 indicating the role never did so, 5 being always and 6 being not applicable.
The problem is, the selections were mutually-exclusive.
In other words, even if I believed all roles were equal in ethics leadership, I had to rank them against each other even though that was not the purpose of the question. And I would have to pick one as completely noncompliant, even if that were untrue.
After noticing the next two questions were similarly faulty, I opted out of the survey and sent an email to the organizing group letting them know my findings and feelings. The point is that if the design and mechanisms are fatally flawed, so too will be any analysis of the results.
I haven’t surveyed this scientifically, but I believe I’m seeing a disturbing trend here, as this is not an isolated example. Far too many of the surveys I encounter lately are poorly designed or lacking in some manner, most commonly in a failure to include “Other” options.
Some survey creators try to avoid criticism with disclaimers that their questionnaire is not scientific. CNN is good at that with their daily Quick Vote feature, which consistently pigeonholes respondents into too few choices yet begs off any responsibility with the phrase “This is not a scientific poll“– but which does not appear until after the visitor has answered or opted to simply view results. How convenient.
It’s tempting to dismiss this sort of thing as a mere annoyance until one realizes how many critical decisions are based upon survey results. The phrase “garbage in, garbage out” takes on scary implications here. This is particularly true when legal referendums are concerned; some Propositions are so misleadingly worded, by design or ineptitude, to where voting “No” is actually a vote for the subject but appears to support it (or vice-versa).
It’s this sort of inanity or worse, malfeasance, that disenfranchises a population and fosters distrust of authoritative bodies. If an organization is truly serious about their surveying, it behooves them to not only spend the appropriate effort on design but also on testing… and I’m thinking the latter is getting less and less attention all the time.
Hmmm… I need to see if I can install the WordPress poll plugin and start teaching by example…
EDIT: after I notified the responsible party, the poll was quickly corrected.