Online Polling: The Good, The Bad and The Ugly

“Church decline in England and Wales has not only stopped, but the Church is growing, as Gen Z leads an exciting turnaround in church attendance. Church attendance has risen by 50 per cent over the last six years, busting the myth of church decline. The findings come in The Quiet Revival, a survey commissioned by Bible Society and conducted by YouGov”.

Online opinion polling, run by large corporations such as YouGov or Ipsos, is an increasingly popular research method that’s developed over the past 25 years with the spread of online connectivity – and like most research methods it’s got its good and its bad points.

Possibly uniquely, however, it has an ugly little secret that’s worth knowing if you’re looking to go that extra evaluative mile that’s not (yet) mentioned in the textbooks…

The Good…

As you might expect, online polling – when carried-out by relatively large, reputable, organisations – has some significant strengths:

Sampling: Because of the data they hold about respondents when they sign-up to the process (about which more in a moment), online polls can easily create stratified samples that accurately reflect whatever population criteria you need for your sample: typically, characteristics like income, age, gender, ethnicity and the like.

Targeting: Depending on the kinds of data the polling organisation collects about respondents, it’s possible for researchers to reach relatively small or niche social groups more easily and cheaply than would be possible using traditional sampling methods.

Cost: Depending on the size and scope of the research, online polling can be far cheaper than traditional forms, such as structured interviews or postal questionnaires.

Response Rate: One of the significant weaknesses a postal questionnaire is a low response rate – something that, in turn, affects the accuracy of your sample. With an online poll you’re always  guaranteed a 100% response rate.

The Bad…

Similarly, online polling, just like all research methods, has its weaknesses. With large polling companies the main problem is that the samples used effectively “select themselves” in three main ways:

• respondents have to apply to and join a polling organisation – and since only those accepted as members can participate in a poll this effectively excludes the vast majority of the population who never bothered to apply. Samples, in this respect, are always non-probabilistic – they can never be truly random. And this opens-up the very real possibility of the poll becoming biased.

• each time a poll is commissioned, respondents are invited to participate. The polling organisation, in other words, selects those respondents it (or, more-correctly, its selection algorithm) believes will be most-interested in answering questions on a topic – and this leads to a further possible level of self-selected bias: some or all of your sample may have a personal interest in the poll. This further level of self-selection leads to more questions about how random the sample can be (and means that things can quickly turn ugly…).

• respondents are paid for the polls they take. They have a financial incentive to participate – and since payments are usually quite small there’s an incentive to take as many polls as possible over a short span of time – and respondents taking lots of different polls quickly are probably less-likely to treat them as seriously as the people paying for the poll might like.

Taken together, the fact respondents effectively select themselves (rather than being randomly selected by the researcher) means the samples for online polls are always likely to be unrepresentative and therefore an unreliable guide to public opinion.

While this is hugely-important drawback – and one reason why we should treat the findings of online polls with great caution – it leads to an even more-significant problem…

The Ugly…

While the kinds of things to which we’ve just referred tell us something about the structural strengths and weaknesses of online opinion polls, one general assumption about this type of research method (online and offline questionnaires, structured and unstructured interviews) is that, by-and-large, respondents tell the truth.

In the case of interviews this assumption tends to hold broadly true: most people find it difficult to lie in these face-to-face situations. Similarly, with postal questionnaires the incentive to lie is probably far less than the incentive to simply not return the questionnaire in the first place…

Recent research by Kennedy et.al. (2024) has however shown this is not the case with online polling. They argue there are significant incentives for respondents to lie – and a particular segment of the population, young men, are much more prone to this behaviour than other groups. As they put it “young opt-in respondents are prone to saying ‘Yes’ no matter what is asked”.

To demonstrate this propensity, Kennedy et. al. developed a simple experiment involving 14 online questions asked of 569 American adults. The only difference between this poll and the kinds of polls these adults were used to taking was that the researchers “selected questions for which a ‘Yes’ answer is not credible”.

Respondents were, for example, asked: “Are you licensed to operate a class SSGN submarine?” and given a choice of “Yes or No”.

Despite the fact “A class SSGN is a nuclear-powered U.S. naval submarine equipped with cruise missiles, of which there are only four in operation…the share of U.S. adults qualified to operate such a vessel is approximately 0%”, nearly 6% of the total survey claimed to hold such a license.

Breaking this down further the researchers note “the incidence of this bogus claim was particularly high among Hispanics (24%) and those under age 30 (12%)”. 

A significant percentage of the sample, in other words, claimed to have done something they could not possibly have done.

A second type of question the researchers used was a battery type that gave respondents a list of activities and asked them to check (tick) all they had done in the previous week.

One list, for example, “included two common activities (watched TV, read a book) and four extraordinarily uncommon activities: purchased a private jet, climbed a peak in the Karakoram Mountains, learned to cook halušky, and played jai alai”. 

Once again “The share reporting at least one extremely uncommon activity in the past week…was significantly higher among those age 18 to 29 than those age 30 and over”. As with the previous type of question Kennedy et.al. found some groups were “prone to over-reporting affirmatively that they have some characteristic when that is simply not possible in the aggregate numbers  observed”.

The researchers, you probably won’t be too surprised to learn, concluded that “commercial opt-in respondents in these subgroups are prone to giving bogus answers”.

The important takeaway here is that some groups – particularly young men in the 18 – 30 age group – fail to tell the truth when answering questions in online opinion polls that involve non-probability‑based samples (samples that are not drawn by random selection).

The fact the researchers found this problem was not present in probability-based samples suggests the propensity to provide bogus statements is an inherent weakness of online polls.

And Finally…

In the unlikely event you were wondering how this all relates to the “Christian Revival” argument noted at the start, the group that claimed the highest increase in church-going over the past 6 six years was – it might not come as any great surprise – young men in the 18 – 30 age group…

And if you’re interested in a more-comprehensive breakdown of exactly why the Bible Society poll and claims (“men are now more religious than women”…) don’t stand-up to closer inspection, this article does a pretty good job of explaining it.