Hi! Last time we met up, we talked about some more bad news for Christianity. Namely, Nones have surged forward to become the nation’s biggest religious group–barely, but still. In response, evangelicals made themselves dizzy spinning that data. A few of those spin-doctors have referred to a study released by LifeWay shortly beforehand. They clearly thought that this study painted a rosy portrait. However, it doesn’t accomplish anything of the sort. In fact, this study demonstrates exactly why we should not trust any studies designed, led, or run by evangelical groups. And today, I’ll show you what I mean by that strong statement.
The Study Itself.
“I love solid research. This new release is absolutely incredible.”
Thom Rainer, if he does say so himself
First, allow me to whisk through some basic facts about this story.
LifeWay functions as the propaganda arm of the Southern Baptist Convention (SBC). In that capacity, the group publishes SBC-friendly materials across various media types. Its leaders also create, run, and interpret various studies regarding topics of interest to members of the denomination.
Recently, LifeWay teamed up with a survey house huckster group called Exponential to run a survey of Protestant churches. They sought to learn how many of these churches were growing, shrinking, or staying stagnant. They also wanted to know of any trends regarding church growth across Protestantism. LifeWay titled this study “Becoming Five Multiplication Study,” which I’m abbreviating to “Becoming Five.” It’s based on a new buzzword, multiplication, floating around evangelicalism these past few years.
You can find and download the full study here. Here’s the LifeWay writeup of the study. Lastly, here’s Thom Rainer’s press release about it.
LifeWay’s Survey Is Awesome, Claims LifeWay’s Salesman-in-Chief.
Thom Rainer, just inches away from a retirement he can probably taste by now, tries to get out in front of potential criticisms by front-loading a press release about one of LifeWay’s new surveys with praise. It’s “solid.”
Y’all, even the “statistical nerds” declare that it’s “very accurate!”
(Why do Christians seem like they always trust anything a salesperson says if he happens to share their religious affiliation?)
What the Survey Found.
LifeWay interviewed 1000 pastors by phone about how their churches had fared over the past three years. Mostly. Some questions applied to the past three years, yes. Others asked only about the past year, which was 2017-2018.
Church Growth:
- 28% said their church attendance had declined 6% or more compared to three years ago.
- 33% said attendance had remained within 5% of their numbers from three years ago.
- 39% said it had grown 6% or more in that period.
- 46% said member donations had either decreased or stagnated over the last year.
We are not ever told exactly why the survey designers chose those timeframes, nor exactly why they settled on 5% or 6% as the line-in-the-sand marking a church as growing or failing.
Attendance Numbers:
- 57% of the respondents’ churches had fewer than 100 people attending services every Sunday…
- … a figure which includes 21% who have fewer than 50 attendees.
- 11% had an average of 250+ attendees.
The survey designers don’t share how they decided to set lines between church populations where they did.
Also, it seems weird for them to lump all the 250+ churches into one category. A 250-person church seems like it’d have little in common with a 5000-person one, but here we are. We have no idea, either, how many of those huge churches they signed up for the survey.
Some Interesting Highlights.
Highlights:
- Of the responding pastors aged between 18-44, 55% said their churches experienced growth. But only 33% of the 45+ crowd said this. We are not told what percentage of pastors fit into either age group.
- 42% of evangelical churches claimed growth, while only 34% of mainline churches did. In fact, it kinda sounds like the more extremist the denomination, the more likely that denomination’s pastors were to claim growth.
- Of the teeny-tiny churches with fewer than 50 attendees each week, only 23% claimed growth. But 59% of the biggest churches, the ones with 250+ attendees, claimed growth.
The report doesn’t tell us if any of those teeny-tiny churches are actually church plants, which is to say brand-new churches sprouted by existing congregations (similarly to how European governments colonized America).
And the Big Kahuna Burger, Conversions:
- 54% of pastors had fewer than 10 people who “indicated a new commitment to Christ as Savior.”
- 8% of pastors said they’d seen none of these “new commitments.”
One stat that might confuse outsiders to SBC culture is conversions per existing members. For years, they’ve fretted about this rate. It still shows up in discussions of their baptism drought. It’s also a line item in every SBC Annual Report under “Ratio of Baptisms.” Around 2012, they panicked when it got worse than 1 baptism per 50 existing members. This study mentions that ratio–sort of–as I’ll show you in a moment.
The Impulse to Shoehorn Data Into Narratives.
Evangelicals tend to like stuff they can very easily summarize with arbitrarily-numbered lists. I began noticing that tendency years ago when I ran across the SBC’s absolutely-bizarre writeup about that baptism drought. (You can read the original here.)
No matter the cost to coherence or credibility, that paper’s authors simply had to have exactly five reasons for their decline in baptisms. And then, to balance out that already-arbitrary and surreal list, they then had to create exactly five solutions to those five problems. Naturally, those five solutions sounded even more arbitrary and surreal.
Balanced lists like that soothe evangelicals. The parity appeals to their huge need for structure. The impulse drives them toward the creation of narratives with and around which they can organize data. Unfortunately, their narratives don’t tend to flow naturally from that data, and the results–the organization–don’t tend to give evangelicals good goals for future action.
“Becoming Five” continues that unfortunate tradition.
The Shoehorn Used In This Study.
The study’s name, “Becoming Five,” tells us exactly what narrative its designers have in mind. They’ve created a paradigm here of five levels of church growth.
- Level 1 churches are in decline. They will close if they can’t get it together.
- Level 2 churches have plateaued in growth, with an overall stagnation in numbers.
- As you can expect, Level 3 churches experience some measure of growth in attendance and baptisms.
- Level 4 churches enjoy both the internal growth of Level 3 churches, but also push for the opening of new churches or additions to their existing church properties.
- And Level 5 churches combine Level 4 growth with an intense devotion to growth generally. They plant churches that have already gone on to Level 4 themselves. These churches also emphasize another big buzzword in evangelicalism these past few years, discipling.
Unfortunately, the study’s designers have created distinctions that seem both meaningless and not reflective of reality.
Instead of reflecting reality, these folks seek to tell a story here. And that story will flatter evangelicals, as well as hopefully push their ideology to the forefront as a superior one for churches to adopt because this narratives allows only evangelicals to emerge intact from Christianity’s decline.
Sure, it won’t help any pastors grow their churches or even give them a meaningful bead on what might be preventing their groups from growing.
But hey. Christians can’t have everything. Not anymore.
The Bottom Line.
After collating and booping all their data from the phone interviews, LifeWay discovered the following:
- 35% of their responding churches sat at level 1.
- Another 35% of churches sat at Level 2.
- And 30% of churches sat at Level 3.
Now, we’re already at 100%, right? But apparently they massaged their numbers and somehow got more than 100. (The writeup doesn’t share how that could happen, but the graph we’ll see shortly says it’s from “rounding.”) That’s how they discovered that:
- 7% of responding churches thrived at Level 4.
- And literally 0% of churches enjoyed Level 5.
Wow.
Is it really a meaningful category for a descriptive survey if literally no respondents fit into it?
Now for the Bad News for LifeWay.
In case this point isn’t crystal-clear already, I’m not a statistician. However, neither is Thom Rainer, or for that matter Ed Stetzer, or really any of the other Christians involved in evangelical research that I’ve ever encountered.
Despite my lack of expertise in this area, however, even I can see some very obvious problems with this study.
Our commentariat was similarly unimpressed. For this next part, I draw in part from some stuff our resident quantum wizard had to say. (Infinite Automaton’s entire comment is well worth the read. Our Snek Fren = very much smart.)
The main problem with this study duplicates the problem we see in almost all Christian-designed surveys. It exists to flatter existing Christians and to encourage them to hold strong on their culture wars–and also to exhort them to obey their leaders.
However, these leaders also function as salespeople. Their flocks almost certainly will not be able to perceive this study’s big problems.
But we heathens and dissenters sure can.
The Dangers of Poor Selection Design.
First and foremost, we don’t know enough about how respondents ended up in the survey. The survey writeup claims LifeWay “contacted” by phone 1000 pastors (senior pastors, ministers, or priests) from an array of church sizes and locations around the country. But we don’t know how they decided which people to call from those areas. The report says LifeWay used a random sample list of ministers, but we have no idea how they created their sample.
Nor do we know how many of those pastors answered the phone, nor how many times LifeWay’s people tried to call them. The survey simply says LifeWay “contacted” them. As Snek Fren puts it, did they just keep contacting pastors till they hit the magic number of 1000? I think that’s what happened, after reading their actual report. And that’s not really very good study design.
We also have no idea how many partial responses they received. For that matter, they don’t even note how many “not sure” or “declined to answer” responses pastors gave them in a single one of their graphics or result writeups.
The writeup also claims LifeWay “weighted” responses by region. We don’t know how they did that. They used a “quota” system to run the survey, but never detail how they designed it.
This study begins with poor selection design, so of course it won’t be very useful.
The Dangers of Self-Reporting Dishonesty.
Second and just as concerning, LifeWay conducted this survey by phone and relied entirely on self-reports.
They had no way to double-check anything their respondents said. Unfortunately, we know Christians lie. They lie at the drop of a hat, especially when the lie serves their greater interests or protects their egos. And they won’t even feel guilty doing it.
And really, everybody lies. Didn’t the TV show House teach us that? Obesity researchers have finally figured out that self-reports can’t obtain the objective data they need about people’s eating habits. However, religion researchers still haven’t worked out that cosmic truth.
Any time people fear coming up short somehow in others’ opinions, they’re going to lie about it. So self-reports about touchy subjects are all but useless. All you’ll get in cases like these is aspirational answers–or ones that hide respondents’ inner shame, or that reinforce their self-delusions.
LifeWay’s survey designers here think they’ve crafted questions that limit such answers, but the list itself doesn’t seem like it’d accomplish that goal. Snek Fren sure wasn’t impressed.
Oops. So LifeWay’s survey design doesn’t eliminate the dishonesty we expect regarding a very sensitive topic.
The Dangers of Vague Terminology.
Now we arrive at the main “sin” of this LifeWay report:
Its total lack of concrete, objective definitions and terms renders it useless.
On page 7 of the full report, for example, we see how they decided what churches performed at Level 1 (Declining). These churches suffer from:
- Declining attendance over three years;
- Declining income over the past year;
- Projected declines in spending over the following year;
- Decreases in the number of paid staff they employ;
- Fewer than ten people who “indicate a new commitment to Jesus Christ as Savior in the past 12 months.”
Alas, most of this list consists of unfounded assertions.
We can certainly understand how declines in attendance could cause a lack of church growth. But the report simply takes it as read that declining donations and spending/staffing levels indicate a dying church. It’s possible, but they don’t show their work there.
And then we’ve got that whole “new commitments” twaddle.
The Whole “New Commitments” Twaddle.
The survey designers never explain what a “new commitment” means. They also don’t provide any objective reason for setting the bar where they do on “new commitments.”
Overall, the survey designers express this metric in terms of “new commitments” per 100 existing church members. That’s not a terrible idea, in and of itself. A teeny-tiny church of 10 or 20 people might be growing just fine if they only see three or four “new commitments,” whatever those are, in a year. But a church of 500 seeing 10 “new commitments” would actually be growing way more slowly.
In fact, that whole idea, in a nutshell, is the sentiment behind the SBC’s “Ratio of Baptisms” metric. It’s an expression of how effective the group is at recruitment. But aside from that burst of wisdom, everything else about “new commitments” remains vague.
For example, we lack information about who these “new commitments” represent. Are they fresh conversions from a state of non-belief? Or are they people getting re-baptized? Or are they the children of existing members? The designers never tell us. Nor do the survey designers share how a “new commitment” translates into recruitment numbers or retention rates.
Weird Fussing-About With Numbers.
This survey did some really funny stuff around that “new commitments” metric. Here is the table:
See anything weird about it?
Namely, see how it arranges responses?
- 35% of churches saw 0-4 “new commitments” per 100 attendees.
- 32% saw 5-9.
- 26% saw 10-24.
- 8% saw 25 or more.
Why are the first two categories arranged by 5s, but the third goes by 15 and the last just lumps everything past 25 together? Is it because if they’d been consistent they wouldn’t have ended up with a 1/3 split for the first three categories like they finagled elsewhere? In fact, if they’d gone from 0-9 to 10-19 to 20+, wouldn’t those graphs look downright terrible for Christianity?
Then there’s this: I know from other reading that church plants often grow fairly quickly compared to established churches. Interested Christians switch over to them from the whole local area, though many of these efforts fold within a few years.
With that in mind, look again at the last category. 25+ recruits per 100 members means that for every four members, one person expressed a “new commitment.”
Assuming a “new commitment” equals a successful recruitment attempt, that figure sounds a lot more like a church plant’s dynamic than a 100-member country church’s. Indeed, in LifeWay’s writeup we learn that 46% of those teeny-tiny under-50 churches hit 10 commitments per 100 attendees, while only 18% of the 250+ churches did.
So this graph looks dramatic, but it’s not very useful.
The Narrative of Evangelical Growth.
The survey questions themselves remind me a great deal of those medieval “forms of confession.” The questions here offer a great deal of insight as to what their evangelical question-askers think a growing church requires–Question 8 in particular.
Unsurprisingly, a survey designed by evangelicals for evangelical consumption discovers that evangelical church practices seem to result in churches that grow according to metrics set by evangelicals.
Wow! Who’d’a thunk?
And yet evangelical churches close too. All these activities may do is give a church a slightly better chance of survival–because gods have nothing to do with churches’ success.
The Reframing Game.
Remember Karl Vaters from the last post? In 2017, he wrote a cry-from-the-heart begging his fellow Christians to please quit writing listicles about church growth that consisted of blaming the pastors like him leading struggling churches like his.
Over the ensuing two years, nothing changed about his peers’ love of these listicles or their tendency to blame pastors like him for their churches’ lack of growth. Not only that, but Vaters himself didn’t change either.
In fact, this guy popped up with a listicle trying to reframe this survey as good news, or at least news that isn’t quite as bad as it sounds.
I. Can’t. EVEN.
This survey, despite Thom Rainer’s vast admiration for it, fails in so many ways to provide an accurate picture of churches that decline or grow–or why one might live while others only scrabble for years and then die. It fails to provide any way to predict failure or success, and worse than that, fails to provide an objective list of tasks church leaders can employ to help make their church one of the few that succeeds in growing. But it sure affirms evangelicals’ strategies!
Basically, y’all, this survey really represents the most evangelical thing evangelicals have done in a while!
NEXT UP: The difference between men and women, apparently–at least as very terrible Christians see it. See you next time!
Please Support What I Do!
Come join us on Facebook, Tumblr, Pinterest, Twitter, and our forum at rolltodisbelieve.com! (Also Instagram, where I mostly post cat pictures.)
If you like what you see, I would love to have your support. My PayPal is captain_cassidy@yahoo.com (that’s an underscore in there) for one-time tips. I also welcome monthly patrons via Patreon with Roll to Disbelieve. You can also support the blog through my Amazon Affiliate link! Thanks!
If I get anything wrong about my analyses, please for the love of kittens feel free to tell me so.