Weighting decides the answer, and its assumptions are not in the stated error

The argument

The second objection concerns what happens after the responses are collected. Because no modern sample resembles the population it is meant to represent, the raw answers cannot be reported. They must be adjusted, and critics argue that the adjustment, rather than the data, is what produces the published result. Two decisions do most of the work. The first is which population the sample is being matched to, and on which characteristics. The second, in election polling, is harder: the relevant population is the set of people who will actually vote, which does not exist at the time of asking and has to be modelled. The pollster must assume how the electorate will be composed, and that assumption is an estimate of the thing the poll is supposed to be informing. Critics stress that these choices are defensible rather than arbitrary, and that this is what makes the problem serious. Reasonable analysts working from identical raw responses, differing only in their weighting scheme and their model of who will turn out, can arrive at materially different leads. The decisive point is what the published uncertainty covers. The stated interval reflects the variability expected from sampling alone. It does not include the uncertainty introduced by the choice of weighting variables, by the turnout model, or by the possibility that the assumed composition is simply wrong, and these are plausibly the largest sources of error in the whole exercise. A reader is therefore shown a precise-looking figure whose dominant uncertainty has been left out of the number attached to it. On this view the problem is not that pollsters exercise judgement, which is unavoidable, but that the result is presented as a measurement when it is substantially a model.

Premises

Counter-arguments

Pollsters reply that the objection argues for better disclosure rather than for distrust. Modelling assumptions can be published, alternative specifications can be reported side by side, and some organisations already release the effect of different turnout scenarios so that readers can see the sensitivity directly. They add that model dependence is a feature of every quantitative estimate about a future population, including economic forecasts, and is not usually taken to make them worthless. The appropriate response, on this view, is to read polls as conditional estimates and to attend to the spread across houses rather than to any single figure.

Rejecting the premises

[Rejecting P3] That different defensible specifications yield different results is an argument for reporting the spread rather than for distrust, and the spread across polling houses is publicly visible to any reader who looks at more than one. [Rejecting P4] Modelling uncertainty is increasingly disclosed rather than concealed, with some organisations publishing the effect of alternative turnout scenarios alongside their headline estimate. [Rejecting C] Model dependence characterises every quantitative estimate about a future population, including economic forecasts, and is not ordinarily held to make such estimates untrustworthy.