Chatbots act as a productivity multiplier for knowledge work

The argument

Several controlled studies measure significant productivity gains from chatbot use in knowledge work. A 2022 GitHub study found that developers using Copilot completed a JavaScript scaffolding task 55 per cent faster than those who did not, with no measurable drop in code quality. A randomised trial run by Ethan Mollick and colleagues at Wharton, conducted with the Boston Consulting Group in 2023, found that consultants using GPT-4 completed 12.2 per cent more tasks on average, 25.1 per cent faster, with 40 per cent higher quality on tasks the model could handle. A separate field study by Erik Brynjolfsson, Danielle Li and Lindsey Raymond at Stanford, also in 2023, found that customer-support agents using a generative-AI assistant resolved 14 per cent more issues per hour, with the largest gains accruing to the least experienced workers. The results have replicated across coding, writing, customer service, translation and basic analysis. Critics rightly note that the gains do not generalise to every task — particularly tasks requiring counterintuitive reasoning or up-to-date facts — but the existence of a frontier of useful applications is not in dispute. A tool that consistently raises throughput or quality on a non-trivial set of common work tasks is, by any ordinary definition, useful. The question is no longer whether chatbots can be productive but where and for whom; the studies tell us the productive zone is large enough to matter.

Premises

Counter-arguments

The studies cited measure narrow tasks with clean success criteria, and the argument generalises past what they establish. The 55 per cent figure came from a single scaffolding exercise rather than sustained work on a real codebase, and subsequent field analyses of production repositories have reported rising code churn and duplication — a quality cost that a completion-time measure does not capture. The consultancy trial is the sharpest problem, because the argument omits its other half. The same experiment included tasks placed deliberately beyond the model's competence, and on those the consultants using it performed *worse* than those without — the jagged-frontier finding the authors themselves emphasised. The mechanism was that users could not tell which side of the frontier they were on and deferred to fluent, confident, wrong output. A tool that raises performance where it is capable and lowers it where it is not is not straightforwardly useful; its value depends on the user's ability to tell the cases apart, which is the very thing the study found they could not do. The support-agent result points somewhere similar: gains concentrated among the least experienced workers, which is consistent with the assistant substituting for training rather than raising the ceiling. And 'useful' as the question poses it is a net judgement. Throughput on tasks within the model's competence is one input to that judgement, not the whole of it.

Rejecting the premises

[Rejecting P1] Improving speed on tasks a tool handles well does not make it useful overall if it degrades performance where it does not, and the cited consultancy trial found exactly that pattern on tasks beyond the model's competence. [Rejecting P2] The measured gains come from short, well-specified tasks with clean success criteria; the coding figure was a single scaffolding exercise, and later analyses of production code report increased churn and duplication.