Can you trust US polls?

The failures are identified, measured and corrected in public

Question: Can you trust US polls?
Position1 of 2›: Yes, you can trust US polls
Argument‹2 of 2: The failures are identified, measured and corrected in public

The argument

The second case for trusting polls concerns how the discipline responds to being wrong, which proponents argue is the only test of reliability that means anything. When polling misses, the industry does not simply assert that it will do better. The misses are decomposed. Analysts examine whether the sample under-covered particular groups, whether non-response varied with political characteristics, whether the model of who would actually vote was mis-specified, and whether late movement rather than measurement was responsible. The findings are published, including by the bodies whose own estimates were wrong, and the methods change in response: sampling frames are altered, new variables are added to the weighting, modes of contact are mixed, and the resulting adjustments are documented so that others can criticise them. Proponents treat this openness about failure as the substance of the case rather than as damage control. The comparison they invite is with the alternatives. Anyone dismissing polls still needs some basis for beliefs about public opinion, and the available substitutes are impressions from crowd sizes, social media volume, the enthusiasm of people who agree with the observer, and the instincts of commentators. None of these publishes a method, states an uncertainty, or can be audited after the fact, and all of them have failure modes that are undocumented because nobody is checking. On this view polling is not trustworthy because it is always close to the mark. It is trustworthy because it is the only instrument in this domain whose errors are measurable, measured, and acted upon, which is what allows its accuracy to improve rather than merely vary.

Premises

[P1]When polls miss, the error is decomposed into identifiable sources such as coverage, differential non-response and mis-specified models of who will vote. [P2] Those findings are published, including by the organisations whose own estimates were wrong, and methods are revised in response. [P3] An instrument whose failure modes are documented and corrected can improve, whereas one whose errors are never measured can only vary. [P4] The available alternatives for judging public opinion publish no method, state no uncertainty and cannot be audited after the event. [C] Polls can therefore be trusted as the most reliable instrument available in this domain, because they are the only one whose errors are measured and corrected in public.

Counter-arguments

Critics reply that self-correction is only a virtue if it works, and that the record shows the same categories of error recurring after each round of revision rather than diminishing. A discipline can publish diagnoses indefinitely without converging. They add that the comparison with impressionistic alternatives is a false choice: the honest option is to hold weaker beliefs about public opinion rather than to accept a flawed instrument because nothing better exists. Proponents, on this view, are defending the integrity of the profession rather than the accuracy of its estimates, and those are different claims.

Rejecting the premises

[Rejecting P2] Publishing a diagnosis is not the same as fixing the fault, and the same categories of error have recurred after successive rounds of methodological revision rather than diminishing. [Rejecting P4] The comparison with impressionistic alternatives is a false choice, because the honest response to an unreliable instrument is to hold weaker beliefs about public opinion rather than to accept the instrument for want of a better one. [Rejecting C] The argument establishes the professional integrity of the field rather than the accuracy of its estimates, and trust was being claimed for the estimates.