Non-response has broken the link between the sample and the population

The argument

The technical case against trusting polls does not rest on any individual miss. It rests on the collapse of the condition that made survey sampling work in the first place. Random sampling delivers an unbiased estimate only if whether a selected person is reached and agrees to answer is independent of what is being measured. When nearly everyone contacted took part, that assumption was close enough to true to be ignored. It can no longer be ignored, because the share of contacted people who agree to be surveyed has fallen to a small fraction of those approached. Critics argue that the remaining respondents are not a random subset of the public but a self-selected one, defined by willingness to spend time answering questions from a stranger about politics. That willingness is not politically neutral. It is plausibly related to political interest, to civic engagement, to trust in institutions and in the media, and to whether a person feels their views are respected by the people asking. These are among the very characteristics a political poll exists to measure, which means the selection is correlated with the quantity being estimated. The consequence is the part critics emphasise most. Error of this kind is not reduced by interviewing more people, because every additional respondent is drawn from the same self-selected pool, so a larger sample delivers a more precise estimate of a biased quantity. The reported uncertainty, which is calculated as though the sample were random, therefore understates the real uncertainty by an unknown amount. On this account the published figures are not estimates with a known error but estimates with an unmeasured one, and that is not a basis for trust.

Premises

Counter-arguments

Pollsters answer that this describes a problem they have spent years addressing rather than one they have ignored. Weighting on variables associated with response propensity, including political and attitudinal measures rather than demographics alone, is designed precisely to correct differential participation, and mixed modes of contact reach people who decline one channel but accept another. They add that the argument proves too much: if self-selection were as severe as claimed, estimates would be wildly and consistently off, whereas most are close and the misses are modest. A bias that is small and partially correctable is not a reason to discard the instrument.

Rejecting the premises

[Rejecting P3] Response propensity is addressed directly by weighting on political and attitudinal variables rather than demographics alone, and by mixing modes of contact to reach people who decline one channel but accept another. [Rejecting P4] The claim that sample size cannot help assumes the bias is uncorrectable, whereas corrections aimed at participation itself change the composition of the weighted sample rather than merely enlarging it. [Rejecting C] If self-selection were as severe as this, estimates would be wildly and consistently wrong, whereas most fall close to the outcome and the misses are modest, which is evidence of a small and partially corrected bias.