Confabulation makes chatbot output structurally untrustworthy

The argument

Large language models do not retrieve facts; they predict plausible token sequences. The result is a fluent text generator with no internal mechanism for distinguishing what it knows from what it has merely interpolated. In practice the output is often correct, and often almost-correct in ways that are hard to detect without already knowing the answer. A 2024 study from Stanford's RegLab found that purpose-built legal chatbots hallucinated case law in 17 to 33 per cent of queries, and that general-purpose models such as GPT-4 hallucinated in 58 to 82 per cent of legal queries. Lawyers, doctors and journalists have been publicly sanctioned, or have publicly retracted work, after citing chatbot-fabricated sources. The pattern is not a bug being patched away by the next model release. Emily Bender and her colleagues described it in 2021 as the defining property of a stochastic parrot: convincing form without verified content. Because the output is fluent, the user is structurally biased toward trusting it, and verification — when it happens at all — often takes longer than producing the answer from a trusted source from scratch. A tool that requires the user to redo the work in order to check it is at best a slow autocomplete; at worst it is a confident liar with a perfect manner.

Premises

Counter-arguments

The verification-cost argument generalises from an unusually demanding domain. Legal research is close to the worst case for these systems: it depends on precise citation of real authorities, a fabricated case is catastrophic and immediately visible, and checking is expensive. Most everyday use has a different cost structure. Drafting text the user then edits, translating into a language the user reads, summarising a document the user already has, or generating code that either compiles and passes tests or does not — in each case verification is cheap, immediate, or performed by something other than the user's memory. The hallucination figures are also configuration-dependent rather than intrinsic. Rates vary enormously according to whether a system answers from parameters alone or is grounded in retrieved documents with links back to sources, and the numbers cited were measured on particular tools and model generations. Treating them as a fixed property of the technology reads a snapshot as a constant. 'No native mechanism to mark what they do not know' is likewise stronger than the evidence supports. Calibration is imperfect, but it is measurable rather than absent, and systems can be and routinely are configured to abstain, express uncertainty and cite sources — which is a design choice about deployment rather than an impossibility. Finally, the conclusion concerns net harm, which requires weighing benefits the argument never counts. It establishes a real and serious failure mode; establishing that the failures outweigh everything else is a further step it does not take.

Rejecting the premises

[Rejecting P1] Verification cost varies by task: editing drafted text, reading a translation, or running generated code against tests are cheap or self-verifying, so the burden the argument describes is not general. [Rejecting P2] Hallucination rates depend heavily on whether a system is grounded in retrieved sources and on the model generation measured, and abstention, uncertainty expression and citation are available design choices rather than impossibilities.