a survey pop-up form
Reading Time: 7 minutes Not the same one, but similar.
Reading Time: 7 minutes

Hi and welcome back! Yesterday, I showed you a new Gallup poll that bodes very poorly for American Christianity. And it made me remember other times I’ve seen Christians ask questions in very leading ways on their polls and surveys. How questioners phrase their questions really matters — and quite a few Christians very obviously do not want to gather accurate information.

yes, he's pretending it was contemporary
Pieter Brueghel the Elder, the Census at Bethlehem, 1566. Christians’ affection for disastrously-inept questions began early.

(Due to a death in my husband’s family, I’m moving with this topic today. Tomorrow, we will have our Easter Egg-Stravaganza. And then, I want to devote some good quality time to the topic of churchless believers, because in plotting out the topic it became very apparent very quickly that there’s a lot of ground to cover.)

This Topic’s Dramatic Origin Story: The Questions They Asked.

A while back, I was reading Christianity Today (CT). And during that visit, I got one of those survey request pop-ups. You know the kind? It didn’t look exactly like this one, but it was similar:

a survey pop-up form
Not the same one, but similar.

Christianity Today said it’d super help them out if I took it. I’m a sucker for any opportunity to give my opinion, so I thought, why not? Maybe I could help CT get acquainted with their heathen readers. Wouldn’t that be a hoot!

Now, I don’t hate CT at all. For a Christian news/opinion site, it meets expectations fairly well and is useful to me in my work. They’re not the worst evangelicals out there.

But this situation got more ridiculous by the question. Very rapidly, I got frustrated with this survey.

If I had to make a guess, I’d say that CT put this survey together specifically to bolster a major new campaign to gain new advertisers. Like maybe they could brandish it and say “Hey! Lookie this! Don’t you want to advertise on a site where 99.999% of survey respondents say we changed their life forever and they’d never go anywhere but here for their baby-Jesus reading needs?”

They didn’t really want accurate answers from me about opportunities for improvement. They wanted numbers they could plug into emails to advertisers. 

If someone doesn’t ask the right questionsthey’re going to get back unhelpful answers. But these guys didn’t even want helpful answers. They just wanted answers their potential advertisers would like. Or maybe it was the birthday of the site’s owner. I don’t know. They wanted to make sure someone knew how adored CT was, even if they had to strong-arm those results.

How to Design a Terrible Survey.

Bad survey questions can easily muddy your data and derail your business decisions.

Get Feedback

Man, I wish I’d screencapped that entire exercise in hilarity. In essence, CT wanted me to love-bomb them. They wanted me to tell them, via answers to multiple-non-choice questions, how much I lurrrrved CT and how much CT had totes for realsies changed my life and my understanding of Jesus.

At no point did any question give an option that came close to capturing my real opinion about literally any aspect of CT’s operations — even if that opinion would have been fairly positive.

Questions were closed to the point of pointlessness, each one leading survey-takers by the nose to the outcome the magazine’s owners clearly wanted to achieve. One question asked me to assign three adjectives to the site; all were incredibly positive. In fact, they had about three of those questions, and all of the adjectives were positive to the point of confusing me: Like who even knows, anymore? Maybe I’m the nutjob and Christians really do consider their news sites to be exactly like close family members.

Very little room even existed to make suggestions about improving the site or its articles/posts. In fact, the survey designers had clearly never even considered that a reader might not actually be a true-blue evangelical Christian. I can’t remember a single place that allowed me to assert anything but stellar perfection regarding their site.

And then this survey, as poorly-made as it was, got me remembering other times when the form of a question really mattered. 

Questions for Creationists.

Back in 2013, the National Center for Science Education (NCSE) got to wondering something: just how many Creationists are there in the United States?

Gallup (Oh wow, it’s them again! Hi, Gallup!) has been asking Americans for decades about their beliefs. Creationism is on their list of topics. They leave out the exact buzzwords, but still, it’s a really leading question:

Which of the following statements comes closest to your views on the origin and development of human beings: human beings have evolved over millions of years from other forms of life and God guided this process, human beings have evolved over millions of years from other forms of life, but God had no part in this process, or God created human beings in their present form at one time within the last 10,000 years.

By 2013, Americans had shown themselves quite consistent over the decades in their responses. About 44% of respondents consistently went for Young Earth Creationism (YEC), 37% for some variation on Creationism, and 12% for the correct non-religious option. (BTW, that’s sure changed.)

But NCSE had noticed that other surveys that asked the question in a less-leading way got really different answers. Some surveys asked a leading question and a non-leading question elsewhere, and got dramatically different answers both times. So NCSE’s writer, Josh Rosenau, concluded then that maybe only 10% of Americans went for YEC. He thought that perhaps another quarter of Americans tried to split the difference between reality and beliefs with a variation of Creationism.

The way science questions get asked matters ginormously. If a question brushes up against Christians’ antiprocess shields, they’ll have a lot of trouble working their way through it. They might feel they must defend the faith rather than use their brain’s science-thinkie-thingie compartments.

Questions About Affiliation.

But science isn’t the only potential threat to a Christian’s antiprocess shields. There’s also tradition, affiliation, feelings of belonging-ness, group identity, and more that can interfere with someone’s ability to think through a question and then deliver an answer that fits how they really feel about the matter.

Countries that don’t erect walls between religion and government have to cope with this exact problem all the time. For example, the United Kingdom (UK) asks about religion on its census. They use the answers they get to allocate funds to religious organizations and efforts (like paying for chaplains in hospitals). They also use those answers to determine what they need to do to prevent religious discrimination.

And slowly, people are starting to realize that the UK government might not be getting a truly accurate picture of religious belief and affiliation because of how they ask their religion questions.

The current way the census asks is quite leading: “What is your religion?”

This question assumes that the respondent does indeed have a religion. It may make someone feel they must answer with the name of some religion or other. Nor does this question ask respondents exactly how they express this “religion” — like are they very observant? Or just culturally members of that faith, whatever it is?

The UK bases a lot of decisions on this census.

And they’re not getting accurate answers, so their decisions won’t be optimal and their outcomes won’t be fair for everyone.

How Much Good Questions Matter.

And we know now that their current census question isn’t giving the UK accurate answers. We know this for sure. And we know it because there’s another high-level UK survey, the British Social Attitudes Survey, that asks about religion in a much more accurate way. They ask: “Do you regard yourself as belonging to any particular religion?” followed by “If yes, which?”

And check out how different the answers are between the two surveys:

Compared to the British Social Attitudes Survey, which asks about belonging to a particular religion and has consistently shown since 2013 that between 48 and 53 percent of respondents are non-religious, the 2001 and 2011 censuses put this figure considerably lower at 15 and 25 percent respectively. [source]

That is a huge difference! But the UK doesn’t want to change their awful question. Their census people think they can just campaign a lot to get the word out about what this question means.

(Yeah, sure, of course. Okay. [j-law.gif] Cuz that’s always worked so amazingly well in years past and with other very important topics, right? Right!)

Accurate Questions Are Only Getting More Important.

Ineptly-written questions also figure high on my list of why I don’t put much trust in evangelical Christian-created and -led surveys. It seems like more more often, these Christians just want to end up with a survey that seems to function more as marketing for their various products (or an exhortation to keep Jesus-ing however they’re already Jesus-ing). They don’t really want to produce a real piece of research meant to add to anyone’s understanding of the world.

And maybe that dysfunctional habit toxic Christians have of trying to faux-research their way out of change figures more prominently than most folks think in Christianity’s dramatic decline.

In coming months and years, as Christianity’s decline continues apace, I expect to see some primo hand-waving out of their biggest-name leaders to negate reputable surveys and studies — especially as survey creators learn more about designing accurate questions. I also expect to see some downright hilarious and cringey faux-research out of that crowd to try to compete with their enemies’ better data.

Our world is on the cusp of some big serious changes, and accurately documenting that change is going to become more and more important to the process of predicting and adjusting to it all.

NEXT UP: We’re gonna do just as much to celebrate the Great Jewish Zombie Uprising as people did in Jerusalem in 33CE. I’m mulling a movie review — see this post’s comments if you want to make a suggestion (just please don’t ask for horrorcore like Passion of the Christ). See you tomorrow! Happy Chocolate Bunny Day!

Please Support What I Do!

Come join us on FacebookTumblr, and Twitter! (Also Instagram, where I mostly post cat pictures, and Pinterest, where I sometimes post vintage recipes from my mom’s old recipe box.)

Also please check out our Graceful Atheist podcast interview

If you like what you see, I gratefully welcome your support. Please consider becoming one of my monthly patrons via Patreon with Roll to Disbelieve for as little as $1/month! My PayPal is captain_cassidy@yahoo.com (that’s an underscore in there) for one-time tips.

You can also support this blog at no extra cost to yourself by beginning your Amazon shopping trips with my affiliate link — and, of course, by liking and sharing my posts on social media!

This blog exists because of readers’ support, and I appreciate every single bit of it. Thank you. <3

Avatar photo

ROLL TO DISBELIEVE "Captain Cassidy" is Cassidy McGillicuddy, a Gen Xer and ex-Pentecostal. (The title is metaphorical.) She writes about the intersection of psychology, belief, popular culture, science,...