Useful for drafting, dangerous for decisions

The argument

The same chatbot can be a useful drafting partner and a dangerous oracle depending only on how the user treats its output. For low-stakes generative tasks — brainstorming, outlining, code scaffolding, translation suggestions, first-draft summaries — the cost of an individual error is small and the user is naturally expected to revise the output before relying on it. The chatbot in that mode is doing what a junior assistant would do, more quickly and at lower cost. For decision-grade tasks — medical advice, legal filings, financial analysis, news summaries used to inform real opinions — the same fluent output becomes a hazard. The fluency masks the error and the user, by definition, lacks the bandwidth or the expertise to verify every claim. In 2023 a New York lawyer was sanctioned for filing a brief citing six chatbot-fabricated cases; the cases were plausible, the citations were formatted correctly, and only opposing counsel's verification exposed the fabrication. The lesson is not that chatbots are universally useful or universally harmful; it is that the question 'are chatbots useful' has no answer independent of the use case to which they are applied and the cost of being wrong in that use case.

Premises

Counter-arguments

Critics argue the neat split by stakes is not where reliability actually breaks. Low-stakes drafting is not as safe as implied: a subtle bug in scaffolded code, a mistranslated clause or a plausible-but-wrong outline can propagate precisely because the user, treating it as low-stakes, does not check carefully. Conversely, high-stakes tasks can be made reliable when a verification step is built in — the sanctioned-lawyer case failed not because the task was high-stakes but because no one verified the citations. On this reading the real variable is whether the output is verified, not the category of use. They add that a position which resolves every case into 'it depends on the use case' risks being under-committal — true but close to unfalsifiable, and offering little guidance until the actual determinant (a verification workflow, the checkability of the output) is specified. Advocates of the stronger 'yes' and 'no' positions each argue their side captures the general tendency the 'depends' framing declines to commit to.

Rejecting the premises

[Rejecting P1] Critics argue utility tracks whether output is verified rather than the intrinsic stakes; low-stakes drafting errors also propagate when unchecked, and high-stakes tasks can be reliable when verification is built in. [Rejecting P2] The claim that chatbots are 'reliable enough' for drafting but not decisions draws the line at stakes, but the sanctioned-lawyer example failed for lack of verification, not because the task was high-stakes, so the dividing variable is misidentified.